opened
lmcrazy
username_0
Hi,I want to use sparking water and H2O for data exchange.
I use external model and my env is :
(1)spark 2.2.0
(2)sparking water 2.2
(3)hadoop CDH.5.10 hadoop 2.6
(4)Execution mode  `YARN-cluster`

The program is printed as follows

'''Sparkling Water Context:
 * Sparkling Water Version: 2.2.42
 * H2O name: mlp
 * cluster size: 1
 * list of used nodes:
  (executorId, host, port)
  ------------------------
  (0,172.16.161.114,54321)
  ------------------------

  Open H2O Flow in browser: http://172.16.7.132:54321 (CMD + click in Mac OSX)'''

Sometimes it works. 
![image](https://user-images.githubusercontent.com/9508140/67278137-f8855600-f4fa-11e9-8e5f-3ab3f94a05b9.png)

Sometimes it gets stuck and no output.
![image](https://user-images.githubusercontent.com/9508140/67278202-194dab80-f4fb-11e9-8ea3-8ccceac206dc.png)
the spark water can not connected to the H2O cluster
issue
created
mn-mikke
username_1
Any chance to upgrade Sparkling Water to `3.26.8-2.2` on your side? If the upgrade doesn't help can you send  us full yarn logs? `yarn logs -applicationID <application ID>`
comment
created
lmcrazy
username_0
@username_1 thanks.
I upgrade Sparkling Water to 3.26.8-2.2 and download extended H2O driver for Kluster mode,use ./get-extended-h2o.sh cdh5.10. Get the h2odriver-sw3.26.8-2-cdh5.10-extended.jar.,then run this package on yarn, cmd is  nohup hadoop jar h2odriver-sw3.26.8-2-cdh5.10-extended.jar -jobname mlp -nodes 1 -mapperXmx 6g >h2ocluster1.txt 2>&1 &.

Now my operating environment is as follows:
(1)spark 2.2.0
(2)sparking water 3.26.8-2.2
(3)hadoop CDH.5.10 hadoop 2.6
(4)Execution mode YARN-client

My code is as follows：
'''
object RL {
  //val valuesCnt = 10
  //val partitions = 2
  val cloudName = "mlp"
}

object App {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder()
      .appName("SW516")
      //.config("spark.ext.h2o.external.read.confirmation.timeout", 300)
      .config("spark.sql.autoBroadcastJoinThreshold", -1)
      .config("spark.locality.wait", 30000)
      .getOrCreate()
    val h2oConf = new H2OConf(spark)
      .setExternalClusterMode()
      .setH2OCluster("172.16.161.114", 54321)
      .useManualClusterStart()
      .setClusterSize(1)
      .set("spark.ext.h2o.cloud.representative", "172.16.161.114:54321")
      .setReplDisabled()
      .setCloudName(cloudName)
    val h2oContext = H2OContext.getOrCreate(spark, h2oConf)

    print(h2oContext.h2oNodes) 
    // get hex from External h2o
    val hourDataFrame = h2oContext.asDataFrame("hour.hex", true)
    print("jghgghg")


    // =====================================
    // ============predit=================
    // =====================================
    import spark.implicits._
    val labeledHourRDD = hourDataFrame.select("cnt", "season").rdd.map(_.mkString(",")).map(line => {
      val linearr = line.trim().split(",")
      val linedoublearr = linearr.map(x => x.toDouble)
      val linearrbuff = linedoublearr.toBuffer
      linearrbuff.remove(0)
      val vectorarr = linearrbuff.toArray
      val vector = Vectors.dense(vectorarr)
      Row(linedoublearr(0), vector)
    })

    //    labeledHourRDD.

    val struct = StructType(Array(StructField("label", DoubleType), StructField("features", VectorType)))

    val labeledHourDF = spark.createDataFrame(labeledHourRDD, struct)

    val lr = new LinearRegression().setMaxIter(10).setRegParam(0.3).setElasticNetParam(0.8)

    // Fit the model
    val lrModel = lr.fit(labeledHourDF)

   
    // Print the coefficients and intercept for linear regression
    println(s"Coefficients: ${lrModel.coefficients} Intercept: ${lrModel.intercept}")

    // Summarize the model over the training set and print out some metrics
    val trainingSummary = lrModel.summary
    println(s"numIterations: ${trainingSummary.totalIterations}")
    println(s"objectiveHistory: [${trainingSummary.objectiveHistory.mkString(",")}]")
    trainingSummary.residuals.show()
    println(s"RMSE: ${trainingSummary.rootMeanSquaredError}")
    println(s"r2: ${trainingSummary.r2}")

    // stop
    h2oContext.stop()
  }
}
'''
I tried several times,sometimes it works,but it also gets stuck and the program will not run at this line of code.
‘’‘ print(h2oContext.h2oNodes) ’‘’
![image](https://user-images.githubusercontent.com/9508140/67303316-df919a80-f524-11e9-8e1b-13239a449174.png)

yarn log:
![image](https://user-images.githubusercontent.com/9508140/67303461-15368380-f525-11e9-91c3-d62ec859077d.png)

Yarn task has no log generation.

Thank you for your help！
comment
created
lmcrazy
username_0
@username_1 **When I set the client's log level to debug, output these logs**
`
(base) [root@graph3 spark2_client]# bin/spark-submit --master yarn --deploy-mode client --jars /opt/sparkling-water-3.26.8-2.2/assembly/build/libs/sparkling-water-assembly_2.11-3.26.8-2.2-all.jar --executor-memory 2g --driver-memory 1g --num-executors 1 --class com.yoyosys.sparkling.demo.App /opt/packages/h2o_sparkingwater-1.0-SNAPSHOT-jar-with-dependencies.jar
WARNING: User-defined SPARK_HOME (/home/console//spark2_client) overrides detected (/home/console/spark2_client).
WARNING: Running spark-class from user-defined location.
19/10/23 11:51:43 WARN H2OConf: Using external cluster mode!
10-23 11:51:45.999 172.16.161.115:54321  28925  main      INFO: Sparkling Water version: 3.26.8-2.2
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO: Spark version: 2.2.0.cloudera2
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO: Integrated H2O version: 3.26.0.8
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO: The following Spark configuration is used:
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.ext.h2o.cloud.name,mlp)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.driver.host,172.16.161.115)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.eventLog.enabled,true)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.app.id,application_1569940766178_0058)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.yarn.historyServer.address,graph3:18081)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.yarn.am.memory,512m)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.yarn.secondary.jars,sparkling-water-assembly_2.11-3.26.8-2.2-all.jar)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.driver.memory,1g)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.jars,file:/opt/packages/h2o_sparkingwater-1.0-SNAPSHOT-jar-with-dependencies.jar)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.serializer,org.apache.spark.serializer.KryoSerializer)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.driver.port,35488)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.ext.h2o.client.log.dir,/home/console/spark2_client/h2ologs/application_1569940766178_0058)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.yarn.jars,)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.ext.h2o.repl.enabled,false)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.ext.h2o.client.log.level,DEBUG)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.submit.deployMode,client)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.ext.h2o.client.ignore.SPARK_PUBLIC_DNS,true)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.sql.autoBroadcastJoinThreshold,-1)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.app.name,SW516)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.ui.filters,org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.ext.h2o.backend.cluster.mode,external)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.ext.h2o.external.cluster.size,1)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.ext.h2o.external.start.mode,manual)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.executor.memory,2g)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES,http://graph1:8088/proxy/application_1569940766178_0058,http://graph3:8088/proxy/application_1569940766178_0058)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.ext.h2o.cloud.representative,172.16.161.114:54321)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.executor.id,driver)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS,graph1,graph3)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.broadcast.compress,true)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.executor.extraLibraryPath,/home/console/hadoop_common/lib/native)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.yarn.dist.jars,file:/opt/sparkling-water-3.26.8-2.2/assembly/build/libs/sparkling-water-assembly_2.11-3.26.8-2.2-all.jar)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.eventLog.dir,hdfs://mycluster//spark2/historylogs)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.ext.h2o.client.verbose,true)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.ext.h2o.user.name,root)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.master,yarn)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.driver.extraLibraryPath,/home/console/hadoop_common/lib/native)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.locality.wait,30000)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.ext.h2o.cloud.timeout,160000)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.ext.h2o.cluster.client.retry.timeout,160000)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.executor.instances,1)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.eventLog.compress,true)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.driver.appUIAddress,http://172.16.161.115:4040)
10-23 11:51:46.049 172.16.161.115:54321  28925  main      INFO:     (spark.yarn.am.extraLibraryPath,/home/console/hadoop_common/lib/native)
10-23 11:51:46.050 172.16.161.115:54321  28925  main      INFO:
10-23 11:51:46.050 172.16.161.115:54321  28925  main      INFO: ----- H2O started (client) -----
10-23 11:51:46.050 172.16.161.115:54321  28925  main      INFO: Build git branch: rel-yau
10-23 11:51:46.050 172.16.161.115:54321  28925  main      INFO: Build git hash: 1228756562cdcd0ae86217bc43220be6855ffe9a
10-23 11:51:46.050 172.16.161.115:54321  28925  main      INFO: Build git describe: jenkins-3.26.0.6-49-g1228756
10-23 11:51:46.050 172.16.161.115:54321  28925  main      INFO: Build project version: 3.26.0.8
10-23 11:51:46.050 172.16.161.115:54321  28925  main      INFO: Build age: 5 days
10-23 11:51:46.051 172.16.161.115:54321  28925  main      INFO: Built by: 'jenkins'
10-23 11:51:46.051 172.16.161.115:54321  28925  main      INFO: Built on: '2019-10-18 01:31:23'
10-23 11:51:46.051 172.16.161.115:54321  28925  main      INFO: Found H2O Core extensions: [HiveTableImporter, StackTraceCollector, Watchdog, XGBoost]
10-23 11:51:46.052 172.16.161.115:54321  28925  main      INFO: Processed H2O arguments: [-flatfile, /tmp/spark-daf3d05f-07a6-403a-8b69-7f9607a52660/sparkling-water-678f917a-dfab-4c71-b85c-129db311c3be/flatfile.txt, -name, mlp, -port_offset, 1, -client_disconnect_timeout, 160000, -user_name, root, -log_level, DEBUG, -log_dir, /home/console/spark2_client/h2ologs/application_1569940766178_0058, -baseport, 54321, -ip, 172.16.161.115, -client]
10-23 11:51:46.052 172.16.161.115:54321  28925  main      INFO: Java availableProcessors: 4
10-23 11:51:46.052 172.16.161.115:54321  28925  main      INFO: Java heap totalMemory: 456.5 MB
10-23 11:51:46.052 172.16.161.115:54321  28925  main      INFO: Java heap maxMemory: 910.5 MB
10-23 11:51:46.052 172.16.161.115:54321  28925  main      INFO: Java version: Java 1.8.0_111 (from Oracle Corporation)
10-23 11:51:46.053 172.16.161.115:54321  28925  main      INFO: JVM launch parameters: [-Xmx1g]
10-23 11:51:46.053 172.16.161.115:54321  28925  main      INFO: OS version: Linux 3.10.0-514.el7.x86_64 (amd64)
10-23 11:51:46.053 172.16.161.115:54321  28925  main      INFO: Machine physical memory: 11.58 GB
10-23 11:51:46.053 172.16.161.115:54321  28925  main      INFO: Machine locale: zh_CN
10-23 11:51:46.053 172.16.161.115:54321  28925  main      INFO: X-h2o-cluster-id: 1571802704015
10-23 11:51:46.053 172.16.161.115:54321  28925  main      INFO: User name: 'root'
10-23 11:51:46.053 172.16.161.115:54321  28925  main      INFO: IPv6 stack selected: false
10-23 11:51:46.054 172.16.161.115:54321  28925  main      INFO: Network interface is down: name:virbr0 (virbr0)
10-23 11:51:46.054 172.16.161.115:54321  28925  main      INFO: Network address/interface is not reachable in 150ms: /fe80:0:0:0:7c21:59df:dfd6:86a%ens32/name:ens32 (ens32)
10-23 11:51:46.054 172.16.161.115:54321  28925  main      INFO: Network address/interface is not reachable in 150ms: /fe80:0:0:0:3fdb:c7c7:35c2:7223%ens32/name:ens32 (ens32)
10-23 11:51:46.054 172.16.161.115:54321  28925  main      INFO: Network address/interface is not reachable in 150ms: /fe80:0:0:0:4c0f:6217:14e5:faf4%ens32/name:ens32 (ens32)
10-23 11:51:46.054 172.16.161.115:54321  28925  main      INFO: Possible IP Address: ens32 (ens32), 172.16.161.115
10-23 11:51:46.054 172.16.161.115:54321  28925  main      INFO: Possible IP Address: lo (lo), 0:0:0:0:0:0:0:1%lo
10-23 11:51:46.055 172.16.161.115:54321  28925  main      INFO: Possible IP Address: lo (lo), 127.0.0.1
10-23 11:51:46.055 172.16.161.115:54321  28925  main      INFO: New (possible) client found, timestamp=-28738
10-23 11:51:46.119 172.16.161.115:54321  28925  main      INFO: H2O node running in unencrypted mode.
10-23 11:51:46.123 172.16.161.115:54321  28925  main      INFO: Internal communication uses port: 54322
10-23 11:51:46.123 172.16.161.115:54321  28925  main      INFO: Listening for HTTP and REST traffic on http://172.16.161.115:54321/
10-23 11:51:46.125 172.16.161.115:54321  28925  main      DEBUG: Interface MTU: 1500
10-23 11:51:46.127 172.16.161.115:54321  28925  main      INFO: H2O cloud name: 'mlp' on /172.16.161.115:54321, discovery address /225.1.166.177:57601
10-23 11:51:46.127 172.16.161.115:54321  28925  main      INFO: If you have trouble connecting, try SSH tunneling from your local machine (e.g., via port 55555):
10-23 11:51:46.127 172.16.161.115:54321  28925  main      INFO:   1. Open a terminal and run 'ssh -L 55555:localhost:54321 root@172.16.161.115'
10-23 11:51:46.127 172.16.161.115:54321  28925  main      INFO:   2. Point your browser to http://localhost:55555
10-23 11:51:47.013 172.16.161.115:54321  28925  main      INFO: Log dir: '/home/console/spark2_client/h2ologs/application_1569940766178_0058'
10-23 11:51:47.014 172.16.161.115:54321  28925  main      INFO: Cur dir: '/home/console/spark2_client'
10-23 11:51:47.015 172.16.161.115:54321  28925  main      DEBUG: H2O launch parameters: [ SYSTEM_PROP_PREFIX: sys.ai.h2o., SYSTEM_DEBUG_CORS: sys.ai.h2o.debug.cors, help: false, version: false, name: mlp, flatfile: /tmp/spark-daf3d05f-07a6-403a-8b69-7f9607a52660/sparkling-water-678f917a-dfab-4c71-b85c-129db311c3be/flatfile.txt, ice_root: null, cleaner: false, nthreads: 4, log_dir: /home/console/spark2_client/h2ologs/application_1569940766178_0058, flow_dir: null, disable_web: false, client: true, notify_local: null, hdfs_config: null, hdfs_skip: false, aws_credentials: null, ga_hadoop_ver: null, log_level: DEBUG, random_udp_drop: false, md5skip: false, quiet: false, clientDisconnectTimeout: 10000, features_level: Experimental ]
10-23 11:51:47.016 172.16.161.115:54321  28925  main      DEBUG: Boot class path: /home/console/jdk1.8.0_111/jre/lib/resources.jar:/home/console/jdk1.8.0_111/jre/lib/rt.jar:/home/console/jdk1.8.0_111/jre/lib/sunrsasign.jar:/home/console/jdk1.8.0_111/jre/lib/jsse.jar:/home/console/jdk1.8.0_111/jre/lib/jce.jar:/home/console/jdk1.8.0_111/jre/lib/charsets.jar:/home/console/jdk1.8.0_111/jre/lib/jfr.jar:/home/console/jdk1.8.0_111/jre/classes
10-23 11:51:47.053 172.16.161.115:54321  28925  main      INFO: Distributed HTTP import not available (import from HTTP/HTTPS will be eager)
10-23 11:51:47.058 172.16.161.115:54321  28925  main      INFO: Using HDFS configuration from /home/console/spark2_client/hyconf
10-23 11:51:47.058 172.16.161.115:54321  28925  main      INFO: HDFS subsystem successfully initialized
10-23 11:51:47.074 172.16.161.115:54321  28925  main      INFO: S3 subsystem successfully initialized
10-23 11:51:47.078 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:51:47.154 172.16.161.115:54321  28925  #ogThread DEBUG: Client: /172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:51:47.311 172.16.161.115:54321  28925  main      INFO: GCS subsystem successfully initialized
10-23 11:51:47.311 172.16.161.115:54321  28925  main      INFO: Flow dir: '/root/h2oflows'
10-23 11:51:47.380 172.16.161.115:54321  28925  main      DEBUG: MemGood:   init, (K/V:Zero   + POJO:Zero   + FREE:910.5 MB == MEM_MAX:910.5 MB), desiredKV=610.0 MB NO-OOM
10-23 11:51:47.548 172.16.161.115:54321  28925  main      INFO: Registered parsers: [GUESS, ARFF, XLS, SVMLight, AVRO, ORC, PARQUET, CSV]
10-23 11:51:47.613 172.16.161.115:54321  28925  main      DEBUG: Timing within H2O.main():
10-23 11:51:47.613 172.16.161.115:54321  28925  main      DEBUG:     Args parsing & validation: 4ms
10-23 11:51:47.613 172.16.161.115:54321  28925  main      DEBUG:     Get ICE root: 1ms
10-23 11:51:47.614 172.16.161.115:54321  28925  main      DEBUG:     Print log version: 330ms
10-23 11:51:47.614 172.16.161.115:54321  28925  main      DEBUG:     Detect network address: 11ms
10-23 11:51:47.614 172.16.161.115:54321  28925  main      DEBUG:     Start local node: 1594ms
10-23 11:51:47.614 172.16.161.115:54321  28925  main      DEBUG:     Extensions onLocalNodeStarted(): 3ms
10-23 11:51:47.614 172.16.161.115:54321  28925  main      DEBUG:     RuntimeMxBean: 3ms
10-23 11:51:47.614 172.16.161.115:54321  28925  main      DEBUG:     Initialize persistence layer: 297ms
10-23 11:51:47.614 172.16.161.115:54321  28925  main      DEBUG:     Start network services: 96ms
10-23 11:51:47.614 172.16.161.115:54321  28925  main      DEBUG:     Cloud up: 2ms
10-23 11:51:47.615 172.16.161.115:54321  28925  main      DEBUG:     Start GA: 201ms
10-23 11:51:47.615 172.16.161.115:54321  28925  main      INFO: HiveTableImporter extension initialized
10-23 11:51:47.615 172.16.161.115:54321  28925  main      INFO: StackTraceCollector extension initialized
10-23 11:51:47.615 172.16.161.115:54321  28925  main      INFO: Watchdog extension initialized
10-23 11:51:47.615 172.16.161.115:54321  28925  main      INFO: XGBoost extension initialized
10-23 11:51:47.615 172.16.161.115:54321  28925  main      INFO: Registered 4 core extensions in: 745ms
10-23 11:51:47.616 172.16.161.115:54321  28925  main      INFO: Registered H2O core extensions: [HiveTableImporter, StackTraceCollector, Watchdog, XGBoost]
10-23 11:51:49.204 172.16.161.115:54321  28925  FJ-126-7  DEBUG: Announcing new Cloud Membership: [graph2/172.16.161.114:54321]
10-23 11:51:49.204 172.16.161.115:54321  28925  FJ-126-7  INFO: Cloud of size 1 formed [graph2/172.16.161.114:54321]
10-23 11:51:49.887 172.16.161.115:54321  28925  main      INFO: Found XGBoost backend with library: xgboost4j_gpu
10-23 11:51:49.887 172.16.161.115:54321  28925  main      INFO: XGBoost supported backends: [WITH_GPU, WITH_OMP]
10-23 11:51:50.507 172.16.161.115:54321  28925  main      INFO: Registered: 181 REST APIs in: 2890ms
10-23 11:51:50.507 172.16.161.115:54321  28925  main      INFO: Registered REST API extensions: [XGBoost, Algos, Amazon S3, AutoML, Core V3, TargetEncoder, Core V4]
10-23 11:51:50.508 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsMultinomialV3  (v =  3, impled by hex.ModelMetricsMultinomial)
10-23 11:51:50.570 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.SparkSVMV3.SparkSVMParametersV3  (v =  3, impled by org.apache.spark.ml.spark.models.svm.SparkSVMParameters)
10-23 11:51:50.581 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.DataFrames.H2OFrameIDV3         (v =  3, impled by water.api.DataFrames.IcedH2OFrameID)
10-23 11:51:50.582 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.CoxPHV3                       (v =  3, impled by hex.coxph.CoxPH)
10-23 11:51:50.584 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.MakeGLMModelV3                (v =  3, impled by water.Iced)
10-23 11:51:50.585 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.KeyV3                  (v =  3, impled by water.Iced)
10-23 11:51:50.586 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.RDDs.RDDV3                      (v =  3, impled by water.api.RDDs.IcedRDDInfo)
10-23 11:51:50.587 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.ModelMetricsGLRMV99             (v = 99, impled by hex.glrm.ModelMetricsGLRM)
10-23 11:51:50.588 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.TargetEncoderV3.TargetEncoderParametersV3  (v =  3, impled by ai.h2o.targetencoding.TargetEncoderModel.TargetEncoderParameters)
10-23 11:51:50.671 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.JStackV3               (v =  3, impled by water.util.JStack)
10-23 11:51:50.671 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ImportFilesV3          (v =  3, impled by water.api.schemas3.ImportFilesV3.ImportFiles)
10-23 11:51:50.672 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.TreeStatsV3                   (v =  3, impled by hex.tree.TreeStats)
10-23 11:51:50.673 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.DeepLearningV3.DeepLearningParametersV3  (v =  3, impled by hex.deeplearning.DeepLearningModel.DeepLearningParameters)
10-23 11:51:50.686 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.FieldMetadataV3        (v =  3, impled by water.api.SchemaMetadata.FieldMetadata)
10-23 11:51:50.687 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.InitIDV3               (v =  3, impled by water.Iced)
10-23 11:51:50.721 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GLRMV3.GLRMParametersV3       (v =  3, impled by hex.glrm.GLRMModel.GLRMParameters)
10-23 11:51:50.728 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.RapidsFunctionV3       (v =  3, impled by water.Iced)
10-23 11:51:50.729 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas4.EndpointsListV4        (v =  4, impled by water.Iced)
10-23 11:51:50.729 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.PCAModelV3.PCAModelOutputV3   (v =  3, impled by hex.pca.PCAModel.PCAOutput)
10-23 11:51:50.738 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.CoxPHModelV3.CoxPHModelOutputV3  (v =  3, impled by hex.coxph.CoxPHModel.CoxPHOutput)
10-23 11:51:50.741 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.NaiveBayesModelV3.NaiveBayesModelOutputV3  (v =  3, impled by hex.naivebayes.NaiveBayesModel.NaiveBayesOutput)
10-23 11:51:50.742 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GrepModelV3                   (v =  3, impled by hex.grep.GrepModel)
10-23 11:51:50.744 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.SharedTreeModelV3.SharedTreeModelOutputV3  (v =  3, impled by hex.tree.SharedTreeModel.SharedTreeOutput)
10-23 11:51:50.760 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.HyperSpaceSearchCriteriaV99.CartesianSearchCriteriaV99  (v = 99, impled by hex.grid.HyperSpaceSearchCriteria.CartesianSearchCriteria)
10-23 11:51:50.762 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.NodePersistentStorageV3.NodePersistentStorageEntryV3  (v =  3, impled by water.Iced)
10-23 11:51:50.762 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.DeepWaterV3.DeepWaterParametersV3  (v =  3, impled by hex.deepwater.DeepWaterParameters)
10-23 11:51:50.770 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.RapidsHelpV3.RapidsExpressionV3  (v =  3, impled by water.Iced)
10-23 11:51:50.771 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.RequestSchemaV3        (v =  3, impled by water.Iced)
10-23 11:51:50.772 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.LogsV3                 (v =  3, impled by water.Iced)
10-23 11:51:50.772 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.UnlockKeysV3           (v =  3, impled by water.Iced)
10-23 11:51:50.773 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.SplitFrameV3           (v =  3, impled by hex.SplitFrame)
10-23 11:51:50.773 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.TreeV3                        (v =  3, impled by water.Iced)
10-23 11:51:50.774 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.ExampleV3.ExampleParametersV3  (v =  3, impled by hex.example.ExampleModel.ExampleParameters)
10-23 11:51:50.775 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelBuildersV3        (v =  3, impled by water.Iced)
10-23 11:51:50.776 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.NaiveBayesV3                  (v =  3, impled by hex.naivebayes.NaiveBayes)
10-23 11:51:50.776 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.KeyV3.PartialDependenceKeyV3  (v =  3, impled by water.Iced)
10-23 11:51:50.843 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas4.input.CreateFrameOriginalIV4  (v =  4, impled by hex.createframe.recipes.OriginalCreateFrameRecipe)
10-23 11:51:50.845 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.SharedTreeModelV3             (v =  3, impled by hex.tree.SharedTreeModel)
10-23 11:51:50.846 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas4.EndpointV4             (v =  4, impled by water.api.Route)
10-23 11:51:50.846 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.PartialDependenceV3    (v =  3, impled by hex.PartialDependence)
10-23 11:51:50.848 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.InteractionV3          (v =  3, impled by hex.Interaction)
10-23 11:51:50.848 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas4.ModelInfoV4            (v =  4, impled by hex.ModelBuilder)
10-23 11:51:50.849 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.PSVMModelV3.PSVMModelOutputV3  (v =  3, impled by hex.psvm.PSVMModel.PSVMModelOutput)
10-23 11:51:50.850 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.ModelBuildersHandler.ModelIdV3  (v =  3, impled by water.Iced)
10-23 11:51:50.850 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.KeyV3.ModelKeyV3       (v =  3, impled by water.Iced)
10-23 11:51:50.851 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ImportHiveTableV3      (v =  3, impled by water.Iced)
10-23 11:51:50.851 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GBMModelV3.GBMModelOutputV3   (v =  3, impled by hex.tree.gbm.GBMModel.GBMOutput)
10-23 11:51:50.862 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsBaseV3     (v =  3, impled by hex.ModelMetrics)
10-23 11:51:50.863 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsMultinomialGLMGenericV3  (v =  3, impled by hex.ModelMetricsMultinomialGLMGeneric)
10-23 11:51:50.864 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.KeyV3.JobKeyV3         (v =  3, impled by water.Iced)
10-23 11:51:50.864 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.JStackV3.DStackTraceV3  (v =  3, impled by water.util.JStackCollectorTask.DStackTrace)
10-23 11:51:50.864 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.ExampleV3                     (v =  3, impled by hex.example.Example)
10-23 11:51:50.865 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.RemoveV3               (v =  3, impled by water.Iced)
10-23 11:51:50.865 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GLMRegularizationPathV3       (v =  3, impled by hex.glm.GLMModel.RegularizationPath)
10-23 11:51:50.866 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsOrdinalGLMGenericV3  (v =  3, impled by hex.ModelMetricsOrdinalGLMGeneric)
10-23 11:51:50.867 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.Word2VecV3.Word2VecParametersV3  (v =  3, impled by hex.word2vec.Word2VecModel.Word2VecParameters)
10-23 11:51:50.868 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.Word2VecTransformV3           (v =  3, impled by water.Iced)
10-23 11:51:51.008 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GLMV3                         (v =  3, impled by hex.glm.GLM)
10-23 11:51:51.009 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.RemoveAllV3            (v =  3, impled by water.Iced)
10-23 11:51:51.009 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.RapidsStringV3         (v =  3, impled by water.Iced)
10-23 11:51:51.009 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GrepV3                        (v =  3, impled by hex.grep.Grep)
10-23 11:51:51.010 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.SVDV99                        (v = 99, impled by hex.svd.SVD)
10-23 11:51:51.010 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ImportFilesMultiV3     (v =  3, impled by water.api.schemas3.ImportFilesMultiV3.ImportFilesMulti)
10-23 11:51:51.011 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas99.AssemblyV99           (v = 99, impled by water.Iced)
10-23 11:51:51.012 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ParseSetupV3           (v =  3, impled by water.parser.ParseSetup)
10-23 11:51:51.015 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas4.input.JobIV4           (v =  4, impled by water.Iced)
10-23 11:51:51.016 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GenericModelV3                (v =  3, impled by hex.generic.GenericModel)
10-23 11:51:51.025 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsOrdinalGLMV3  (v =  3, impled by hex.ModelMetricsBinomialGLM.ModelMetricsOrdinalGLM)
10-23 11:51:51.027 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.AggregatorV99                 (v = 99, impled by hex.aggregator.Aggregator)
10-23 11:51:51.027 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas4.ModelsInfoV4           (v =  4, impled by water.Iced)
10-23 11:51:51.028 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.PSVMModelV3                   (v =  3, impled by hex.psvm.PSVMModel)
10-23 11:51:51.028 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.scalaInt.ScalaSessionsV3        (v =  3, impled by water.api.scalaInt.IcedSessions)
10-23 11:51:51.028 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.SVDModelV99                   (v = 99, impled by hex.svd.SVDModel)
10-23 11:51:51.029 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.DeepWaterModelV3              (v =  3, impled by hex.deepwater.DeepWaterModel)
10-23 11:51:51.029 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.NetworkBenchV3         (v =  3, impled by water.init.NetworkBench)
10-23 11:51:51.030 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.scalaInt.ScalaCodeResultV3      (v =  3, impled by water.Iced)
10-23 11:51:51.030 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.NodePersistentStorageV3  (v =  3, impled by water.Iced)
10-23 11:51:51.030 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.TwoDimTableV3.ColumnSpecsBase  (v = -1, impled by water.Iced)
10-23 11:51:51.031 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.FrameV3.ColSpecifierV3  (v =  3, impled by water.fvec.Frame.VecSpecifier)
10-23 11:51:51.031 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.DCTTransformerV3       (v =  3, impled by water.util.DCTTransformer)
10-23 11:51:51.032 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.StringPairV3           (v =  3, impled by hex.StringPair)
10-23 11:51:51.032 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GridSearchSchema              (v = -1, impled by hex.grid.Grid)
10-23 11:51:51.033 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.WaterMeterIoV3         (v =  3, impled by water.util.WaterMeterIo)
10-23 11:51:51.034 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GramV3                        (v =  3, impled by water.Iced)
10-23 11:51:51.034 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsOrdinalGenericV3  (v =  3, impled by hex.ModelMetricsOrdinalGeneric)
10-23 11:51:51.035 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.automl.api.schemas3.AutoMLV99.AutoMLKeyV3  (v =  3, impled by water.Iced)
10-23 11:51:51.035 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.PCAModelV3                    (v =  3, impled by hex.pca.PCAModel)
10-23 11:51:51.068 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.CapabilityEntryV3      (v =  3, impled by water.Iced)
10-23 11:51:51.069 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.SparkSVMV3                    (v =  3, impled by org.apache.spark.ml.spark.models.svm.SparkSVM)
10-23 11:51:51.070 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.PSVMV3                        (v =  3, impled by hex.psvm.PSVM)
10-23 11:51:51.070 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.IsolationForestModelV3        (v =  3, impled by hex.tree.isofor.IsolationForestModel)
10-23 11:51:51.071 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas4.InputSchemaV4          (v =  4, impled by water.Iced)
10-23 11:51:51.071 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.KMeansV3.KMeansParametersV3   (v =  3, impled by hex.kmeans.KMeansModel.KMeansParameters)
10-23 11:51:51.078 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GBMV3.GBMParametersV3         (v =  3, impled by hex.tree.gbm.GBMModel.GBMParameters)
10-23 11:51:51.082 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ParseV3                (v =  3, impled by water.Iced)
10-23 11:51:51.084 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.automl.api.schemas3.AutoMLBuildSpecV99  (v = 99, impled by ai.h2o.automl.AutoMLBuildSpec)
10-23 11:51:51.084 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.DRFModelV3                    (v =  3, impled by hex.tree.drf.DRFModel)
10-23 11:51:51.084 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.TargetEncoderV3               (v =  3, impled by ai.h2o.targetencoding.TargetEncoderBuilder)
10-23 11:51:51.085 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas4.input.CreateFrameSimpleIV4  (v =  4, impled by hex.createframe.recipes.SimpleCreateFrameRecipe)
10-23 11:51:51.088 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.MissingInserterV3      (v =  3, impled by water.util.FrameUtils.MissingInserter)
10-23 11:51:51.088 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.KMeansModelV3                 (v =  3, impled by hex.kmeans.KMeansModel)
10-23 11:51:51.089 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsMultinomialGLMV3  (v =  3, impled by hex.ModelMetricsBinomialGLM.ModelMetricsMultinomialGLM)
10-23 11:51:51.090 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas99.GridsV99              (v = 99, impled by water.api.Grids)
10-23 11:51:51.090 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.KMeansModelV3.KMeansModelOutputV3  (v =  3, impled by hex.kmeans.KMeansModel.KMeansOutput)
10-23 11:51:51.091 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.IsolationForestV3.IsolationForestParametersV3  (v =  3, impled by hex.tree.isofor.IsolationForestModel.IsolationForestParameters)
10-23 11:51:51.093 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.RapidsHelpV3           (v =  3, impled by water.Iced)
10-23 11:51:51.093 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.FindV3                 (v =  3, impled by water.Iced)
10-23 11:51:51.145 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.automl.api.schemas3.LeaderboardsV99  (v = 99, impled by water.automl.api.LeaderboardsHandler.Leaderboards)
10-23 11:51:51.147 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.TimelineV3.IOEvent     (v = -1, impled by water.Iced)
10-23 11:51:51.149 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.ExampleModelV3.ExampleModelOutputV3  (v =  3, impled by hex.example.ExampleModel.ExampleOutput)
10-23 11:51:51.150 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.KeyV3.DecryptionToolKeyV3  (v =  3, impled by water.Iced)
10-23 11:51:51.150 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ImportSQLTableV99      (v = 99, impled by water.Iced)
10-23 11:51:51.151 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.StreamingSchema                 (v = -1, impled by water.Iced)
10-23 11:51:51.151 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelBuilderV3         (v =  3, impled by hex.ModelBuilder)
10-23 11:51:51.151 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.NaiveBayesV3.NaiveBayesParametersV3  (v =  3, impled by hex.naivebayes.NaiveBayesModel.NaiveBayesParameters)
10-23 11:51:51.153 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.RapidsFrameV3          (v =  3, impled by water.Iced)
10-23 11:51:51.157 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.GarbageCollectV3       (v =  3, impled by water.Iced)
10-23 11:51:51.157 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsRegressionV3  (v =  3, impled by hex.ModelMetricsRegression)
10-23 11:51:51.158 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.DeepWaterV3                   (v =  3, impled by hex.deepwater.DeepWater)
10-23 11:51:51.159 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.QuantileV3.QuantileParametersV3  (v =  3, impled by hex.quantile.QuantileModel.QuantileParameters)
10-23 11:51:51.161 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.automl.api.schemas3.AutoMLBuildSpecV99.AutoMLStoppingCriteriaV99  (v = 99, impled by ai.h2o.automl.AutoMLBuildSpec.AutoMLStoppingCriteria)
10-23 11:51:51.164 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GLRMModelV3                   (v =  3, impled by hex.glrm.GLRMModel)
10-23 11:51:51.164 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.KillMinus3V3           (v =  3, impled by water.Iced)
10-23 11:51:51.165 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.IsolationForestModelV3.IsolationForestModelOutputV3  (v =  3, impled by hex.tree.isofor.IsolationForestModel.IsolationForestOutput)
10-23 11:51:51.166 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.AggregatorModelV99.AggregatorModelOutputV99  (v = 99, impled by hex.aggregator.AggregatorModel.AggregatorOutput)
10-23 11:51:51.166 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.HyperSpaceSearchCriteriaV99.RandomDiscreteValueSearchCriteriaV99  (v = 99, impled by hex.grid.HyperSpaceSearchCriteria.RandomDiscreteValueSearchCriteria)
10-23 11:51:51.167 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.RapidsSchemaV3         (v =  3, impled by water.Iced)
10-23 11:51:51.168 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.JobV3                  (v =  3, impled by water.Job)
10-23 11:51:51.169 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.Word2VecModelV3               (v =  3, impled by hex.word2vec.Word2VecModel)
10-23 11:51:51.170 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.automl.api.schemas3.AutoMLBuildSpecV99.AutoMLInputV99  (v = 99, impled by ai.h2o.automl.AutoMLBuildSpec.AutoMLInput)
10-23 11:51:51.171 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.RDDs.RDDsV3                     (v =  3, impled by water.api.RDDs.RDDs)
10-23 11:51:51.171 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.FramesListV3           (v =  3, impled by water.api.FramesHandler.Frames)
10-23 11:51:51.172 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.KeyV3.GridKeyV3        (v =  3, impled by water.Iced)
10-23 11:51:51.172 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.TargetEncoderModelV3          (v =  3, impled by ai.h2o.targetencoding.TargetEncoderModel)
10-23 11:51:51.173 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsBinomialV3  (v =  3, impled by hex.ModelMetricsBinomial)
10-23 11:51:51.174 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsRegressionGLMV3  (v =  3, impled by hex.ModelMetricsRegressionGLM)
10-23 11:51:51.175 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.TargetEncoderTransformParametersV3  (v =  3, impled by ai.h2o.targetencoding.TargetEncoderTransformParameters)
10-23 11:51:51.176 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.TypeaheadV3            (v =  3, impled by water.Iced)
10-23 11:51:51.176 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsRegressionGenericV3  (v =  3, impled by hex.ModelMetricsRegressionGeneric)
10-23 11:51:51.177 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.AboutV3                (v =  3, impled by water.Iced)
10-23 11:51:51.178 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.automl.api.schemas3.EventLogV99     (v = 99, impled by ai.h2o.automl.EventLog)
10-23 11:51:51.178 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas99.RapidsV99             (v = 99, impled by water.Iced)
10-23 11:51:51.179 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.RapidsMapFrameV3       (v =  3, impled by water.Iced)
10-23 11:51:51.180 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ClusteringModelParametersSchemaV3  (v =  3, impled by hex.ClusteringModel.ClusteringParameters)
10-23 11:51:51.191 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.FrameSynopsisV3        (v =  3, impled by water.Iced)
10-23 11:51:51.191 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelParameterSchemaV3  (v =  3, impled by water.Iced)
10-23 11:51:51.192 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.RDDs.RDD2H2OFrameIDV3           (v =  3, impled by water.api.RDDs.IcedRDD2H2OFrameID)
10-23 11:51:51.192 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.RapidsNumbersV3        (v =  3, impled by water.Iced)
10-23 11:51:51.193 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.ExampleModelV3                (v =  3, impled by hex.example.ExampleModel)
10-23 11:51:51.193 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.XGBoostModelV3                (v =  3, impled by hex.tree.xgboost.XGBoostModel)
10-23 11:51:51.239 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GenericModelV3.GenericModelOutputV3  (v =  3, impled by hex.generic.GenericModelOutput)
10-23 11:51:51.240 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelsV3               (v =  3, impled by water.api.ModelsHandler.Models)
10-23 11:51:51.240 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.FrameV3                (v =  3, impled by water.fvec.Frame)
10-23 11:51:51.242 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.automl.api.schemas3.EventLogEntryV99  (v = 99, impled by ai.h2o.automl.EventLogEntry)
10-23 11:51:51.247 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.H2OModelBuilderErrorV3  (v =  3, impled by water.H2OModelBuilderError)
10-23 11:51:51.248 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.KeyV3.FrameKeyV3       (v =  3, impled by water.Iced)
10-23 11:51:51.249 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GLRMModelV3.GLRMModelOutputV3  (v =  3, impled by hex.glrm.GLRMModel.GLRMOutput)
10-23 11:51:51.254 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.DataFrames.DataFramesV3         (v =  3, impled by water.api.DataFrames.DataFrames)
10-23 11:51:51.255 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelOutputSchemaV3    (v =  3, impled by hex.Model.Output)
10-23 11:51:51.255 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.NetworkTestV3          (v =  3, impled by water.init.NetworkTest)
10-23 11:51:51.256 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.CloudV3.NodeV3         (v =  3, impled by water.Iced)
10-23 11:51:51.257 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.RapidsNumberV3         (v =  3, impled by water.Iced)
10-23 11:51:51.258 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.DeepLearningModelV3           (v =  3, impled by hex.deeplearning.DeepLearningModel)
10-23 11:51:51.258 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.JobsV3                 (v =  3, impled by water.Iced)
10-23 11:51:51.259 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.KeyValueV3             (v =  3, impled by hex.KeyValue)
10-23 11:51:51.259 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.util.WaterMeterIo.IoStatsEntry      (v = -1, impled by water.Iced)
10-23 11:51:51.260 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.CoxPHModelV3                  (v =  3, impled by hex.coxph.CoxPHModel)
10-23 11:51:51.260 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ProfilerV3             (v =  3, impled by water.Iced)
10-23 11:51:51.261 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.ModelMetricsAnomalyV3           (v =  3, impled by hex.tree.isofor.ModelMetricsAnomaly)
10-23 11:51:51.261 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.PCAV3.PCAParametersV3         (v =  3, impled by hex.pca.PCAModel.PCAParameters)
10-23 11:51:51.263 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.SparkSVMModelV3.SparkSVMModelOutputV3  (v =  3, impled by org.apache.spark.ml.spark.models.svm.SparkSVMModel.SparkSVMOutput)
10-23 11:51:51.266 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.ModelBuilderSchema            (v = -1, impled by hex.ModelBuilder)
10-23 11:51:51.266 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.DataInfoFrameV3               (v =  3, impled by water.Iced)
10-23 11:51:51.267 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.SchemaV3               (v =  3, impled by water.Iced)
10-23 11:51:51.267 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.StackedEnsembleModelV99.StackedEnsembleModelOutputV99  (v = 99, impled by hex.ensemble.StackedEnsembleModel.StackedEnsembleOutput)
10-23 11:51:51.270 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.automl.api.schemas3.AutoMLBuildSpecV99.AutoMLBuildModelsV99  (v = 99, impled by ai.h2o.automl.AutoMLBuildSpec.AutoMLBuildModels)
10-23 11:51:51.272 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.FramesV3               (v =  3, impled by water.api.FramesHandler.Frames)
10-23 11:51:51.274 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.SharedTreeV3                  (v =  3, impled by hex.tree.SharedTree)
10-23 11:51:51.274 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.IsolationForestV3             (v =  3, impled by hex.tree.isofor.IsolationForest)
10-23 11:51:51.274 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.DeepWaterModelV3.DeepWaterModelOutputV3  (v =  3, impled by hex.deepwater.DeepWaterModelOutput)
10-23 11:51:51.275 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.DeepLearningV3                (v =  3, impled by hex.deeplearning.DeepLearning)
10-23 11:51:51.275 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.Word2VecSynonymsV3            (v =  3, impled by water.Iced)
10-23 11:51:51.277 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.WaterMeterCpuTicksV3   (v =  3, impled by water.util.WaterMeterCpuTicks)
10-23 11:51:51.277 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.DRFV3                         (v =  3, impled by hex.tree.drf.DRF)
10-23 11:51:51.278 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GLMModelV3.GLMModelOutputV3   (v =  3, impled by hex.glm.GLMModel.GLMOutput)
10-23 11:51:51.278 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsOrdinalV3  (v =  3, impled by hex.ModelMetricsOrdinal)
10-23 11:51:51.311 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.KMeansV3                      (v =  3, impled by hex.kmeans.KMeans)
10-23 11:51:51.311 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsBinomialGLMGenericV3  (v =  3, impled by hex.ModelMetricsBinomialGLMGeneric)
10-23 11:51:51.312 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.StackedEnsembleV99            (v = 99, impled by hex.ensemble.StackedEnsemble)
10-23 11:51:51.313 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelExportV3          (v =  3, impled by water.Iced)
10-23 11:51:51.313 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ProfilerNodeV3.ProfilerNodeEntryV3  (v =  3, impled by water.Iced)
10-23 11:51:51.313 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.ModelMetricsHandler.ModelMetricsListSchemaV3  (v =  3, impled by water.api.ModelMetricsHandler.ModelMetricsList)
10-23 11:51:51.316 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.DRFModelV3.DRFModelOutputV3   (v =  3, impled by hex.tree.drf.DRFModel.DRFOutput)
10-23 11:51:51.316 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.TimelineV3.EventV3     (v =  3, impled by water.Iced)
10-23 11:51:51.317 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsMultinomialGenericV3  (v =  3, impled by hex.ModelMetricsMultinomialGeneric)
10-23 11:51:51.317 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelParametersSchemaV3  (v =  3, impled by hex.Model.Parameters)
10-23 11:51:51.317 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.KeyV3.AssemblyKeyV3    (v =  3, impled by water.Iced)
10-23 11:51:51.318 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.scalaInt.ScalaCodeV3            (v =  3, impled by water.api.scalaInt.IcedCode)
10-23 11:51:51.318 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsBinomialGenericV3  (v =  3, impled by hex.ModelMetricsBinomialGeneric)
10-23 11:51:51.319 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.automl.api.schemas3.AutoMLV99       (v = 99, impled by ai.h2o.automl.AutoML)
10-23 11:51:51.319 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.TabulateV3             (v =  3, impled by water.util.Tabulate)
10-23 11:51:51.320 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsClusteringV3  (v =  3, impled by hex.ModelMetricsClustering)
10-23 11:51:51.327 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.RouteV3                (v =  3, impled by water.api.Route)
10-23 11:51:51.328 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.H2OErrorV3             (v =  3, impled by water.H2OError)
10-23 11:51:51.329 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.TimelineV3             (v =  3, impled by water.api.TimelineHandler.Timeline)
10-23 11:51:51.329 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.SharedTreeV3.SharedTreeParametersV3  (v =  3, impled by hex.tree.SharedTreeModel.SharedTreeParameters)
10-23 11:51:51.330 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.PCAV3                         (v =  3, impled by hex.pca.PCA)
10-23 11:51:51.331 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ConfusionMatrixV3      (v =  3, impled by hex.ConfusionMatrix)
10-23 11:51:51.331 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelSynopsisV3        (v =  3, impled by hex.Model)
10-23 11:51:51.331 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.MetadataV3             (v =  3, impled by water.Iced)
10-23 11:51:51.332 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.DecryptionSetupV3      (v =  3, impled by water.parser.DecryptionTool.DecryptionSetup)
10-23 11:51:51.332 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.StackedEnsembleModelV99       (v = 99, impled by hex.ensemble.StackedEnsembleModel)
10-23 11:51:51.333 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.XGBoostV3.XGBoostParametersV3  (v =  3, impled by hex.tree.xgboost.XGBoostModel.XGBoostParameters)
10-23 11:51:51.338 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.DownloadDataV3         (v =  3, impled by water.Iced)
10-23 11:51:51.339 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.XGBoostV3                     (v =  3, impled by hex.tree.xgboost.XGBoost)
10-23 11:51:51.339 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.TwoDimTableV3          (v =  3, impled by water.util.TwoDimTable)
10-23 11:51:51.340 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.TimelineV3.HeartBeatEvent  (v = -1, impled by water.Iced)
10-23 11:51:51.340 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.FrameV3.ColV3          (v =  3, impled by water.fvec.Vec)
10-23 11:51:51.342 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GrepV3.GrepParametersV3       (v =  3, impled by hex.grep.GrepModel.GrepParameters)
10-23 11:51:51.342 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GLRMV3                        (v =  3, impled by hex.glrm.GLRM)
10-23 11:51:51.343 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.ClusteringModelBuilderSchema  (v = -1, impled by hex.ClusteringModelBuilder)
10-23 11:51:51.343 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.automl.api.schemas3.AutoMLBuildSpecV99.AutoMLBuildControlV99  (v = 99, impled by ai.h2o.automl.AutoMLBuildSpec.AutoMLBuildControl)
10-23 11:51:51.344 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.Word2VecV3                    (v =  3, impled by hex.word2vec.Word2Vec)
10-23 11:51:51.344 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GridSchemaV99                 (v = 99, impled by hex.grid.Grid)
10-23 11:51:51.356 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.CapabilitiesV3         (v =  3, impled by water.Iced)
10-23 11:51:51.357 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.PSVMV3.PSVMParametersV3       (v =  3, impled by hex.psvm.PSVMModel.PSVMParameters)
10-23 11:51:51.359 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.ModelMetricsPCAV3               (v =  3, impled by hex.pca.ModelMetricsPCA)
10-23 11:51:51.359 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.SteamMetricsV3         (v =  3, impled by water.api.schemas3.SteamMetricsV3.SteamMetrics)
10-23 11:51:51.359 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelSchemaBaseV3      (v =  3, impled by hex.Model)
10-23 11:51:51.360 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.LogAndEchoV3           (v =  3, impled by water.Iced)
10-23 11:51:51.360 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.scalaInt.ScalaSessionIdV3       (v =  3, impled by water.api.scalaInt.IcedSessionId)
10-23 11:51:51.360 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.H2OFrames.DataFrameIDV3         (v =  3, impled by water.api.H2OFrames.IcedDataFrameID)
10-23 11:51:51.361 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.SVDModelV99.SVDModelOutputV99  (v = 99, impled by hex.svd.SVDModel.SVDOutput)
10-23 11:51:51.361 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.CreateFrameV3          (v =  3, impled by hex.CreateFrame)
10-23 11:51:51.363 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.QuantileV3                    (v =  3, impled by hex.quantile.Quantile)
10-23 11:51:51.363 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.VarImpV3               (v =  3, impled by hex.VarImp)
10-23 11:51:51.363 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.SchemaMetadataV3       (v =  3, impled by water.api.SchemaMetadata)
10-23 11:51:51.364 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.SparkSVMModelV3               (v =  3, impled by org.apache.spark.ml.spark.models.svm.SparkSVMModel)
10-23 11:51:51.364 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GLMV3.GLMParametersV3         (v =  3, impled by hex.glm.GLMModel.GLMParameters)
10-23 11:51:51.406 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas4.ListRequestV4          (v =  4, impled by water.Iced)
10-23 11:51:51.407 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelSchemaV3          (v =  3, impled by hex.Model)
10-23 11:51:51.407 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.automl.api.schemas3.LeaderboardV99  (v = 99, impled by ai.h2o.automl.Leaderboard)
10-23 11:51:51.408 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.RapidsStringsV3        (v =  3, impled by water.Iced)
10-23 11:51:51.408 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GenericV3                     (v =  3, impled by hex.generic.Generic)
10-23 11:51:51.409 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ValidationMessageV3    (v =  3, impled by hex.ModelBuilder.ValidationMessage)
10-23 11:51:51.409 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ProfilerNodeV3         (v =  3, impled by water.Iced)
10-23 11:51:51.410 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.AboutEntryV3           (v =  3, impled by water.Iced)
10-23 11:51:51.410 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.StackedEnsembleV99.StackedEnsembleParametersV99  (v = 99, impled by hex.ensemble.StackedEnsembleModel.StackedEnsembleParameters)
10-23 11:51:51.413 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.CoxPHV3.CoxPHParametersV3     (v =  3, impled by hex.coxph.CoxPHModel.CoxPHParameters)
10-23 11:51:51.415 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsRegressionGLMGenericV3  (v =  3, impled by hex.ModelMetricsRegressionGLMGeneric)
10-23 11:51:51.415 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas4.OutputSchemaV4         (v =  4, impled by water.Iced)
10-23 11:51:51.416 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GenericV3.GenericParametersV3  (v =  3, impled by hex.generic.GenericModelParameters)
10-23 11:51:51.421 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelImportV3          (v =  3, impled by water.Iced)
10-23 11:51:51.422 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsAutoEncoderV3  (v =  3, impled by hex.ModelMetricsAutoEncoder)
10-23 11:51:51.422 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas4.output.JobV4           (v =  4, impled by water.Job)
10-23 11:51:51.424 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.NaiveBayesModelV3             (v =  3, impled by hex.naivebayes.NaiveBayesModel)
10-23 11:51:51.425 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.FrameBaseV3            (v =  3, impled by water.Iced)
10-23 11:51:51.425 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.ModelMetricsHandler.ModelMetricsMakerSchemaV3  (v =  3, impled by water.api.ModelMetricsHandler.ModelMetricsMaker)
10-23 11:51:51.426 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas4.SessionIdV4            (v =  4, impled by water.Iced)
10-23 11:51:51.426 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GBMV3                         (v =  3, impled by hex.tree.gbm.GBM)
10-23 11:51:51.427 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.AggregatorV99.AggregatorParametersV99  (v = 99, impled by hex.aggregator.AggregatorModel.AggregatorParameters)
10-23 11:51:51.428 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GBMModelV3                    (v =  3, impled by hex.tree.gbm.GBMModel)
10-23 11:51:51.428 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.DataFrames.DataFrameV3          (v =  3, impled by water.api.DataFrames.IcedDataFrameInfo)
10-23 11:51:51.429 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.TargetEncoderModelOutputV3    (v =  3, impled by ai.h2o.targetencoding.TargetEncoderModel.TargetEncoderOutput)
10-23 11:51:51.429 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ParseSVMLightV3        (v =  3, impled by water.Iced)
10-23 11:51:51.430 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.SVDV99.SVDParametersV99       (v = 99, impled by hex.svd.SVDModel.SVDParameters)
10-23 11:51:51.431 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.AggregatorModelV99            (v = 99, impled by hex.aggregator.AggregatorModel)
10-23 11:51:51.436 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ShutdownV3             (v =  3, impled by water.api.ShutdownHandler.Shutdown)
10-23 11:51:51.437 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.DRFV3.DRFParametersV3         (v =  3, impled by hex.tree.drf.DRFModel.DRFParameters)
10-23 11:51:51.438 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.XGBoostModelV3.XGBoostModelOutputV3  (v =  3, impled by hex.tree.xgboost.XGBoostOutput)
10-23 11:51:51.439 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.DeepLearningModelV3.DeepLearningModelOutputV3  (v =  3, impled by hex.deeplearning.DeepLearningModel.DeepLearningModelOutput)
10-23 11:51:51.440 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.CloudV3                (v =  3, impled by water.Iced)
10-23 11:51:51.441 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GrepModelV3.GrepModelOutputV3  (v =  3, impled by hex.grep.GrepModel.GrepOutput)
10-23 11:51:51.442 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.TimelineV3.NetworkEvent  (v = -1, impled by water.Iced)
10-23 11:51:51.443 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.GLMModelV3                    (v =  3, impled by hex.glm.GLMModel)
10-23 11:51:51.443 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.persist.PersistS3CredentialsV3      (v =  3, impled by water.Iced)
10-23 11:51:51.444 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.schemas3.ModelMetricsBinomialGLMV3  (v =  3, impled by hex.ModelMetricsBinomialGLM)
10-23 11:51:51.444 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.HyperSpaceSearchCriteriaV99   (v = 99, impled by hex.grid.HyperSpaceSearchCriteria)
10-23 11:51:51.445 172.16.161.115:54321  28925  main      DEBUG: Registering schema: hex.schemas.Word2VecModelV3.Word2VecModelOutputV3  (v =  3, impled by hex.word2vec.Word2VecModel.Word2VecOutput)
10-23 11:51:51.445 172.16.161.115:54321  28925  main      DEBUG: Registering schema: water.api.ModelMetricsSVDV99              (v = 99, impled by hex.svd.SVDModel.ModelMetricsSVD)
10-23 11:51:51.446 172.16.161.115:54321  28925  main      INFO: Registered: 280 schemas in 938ms
10-23 11:51:51.446 172.16.161.115:54321  28925  main      INFO: H2O started in 7122ms
10-23 11:51:51.446 172.16.161.115:54321  28925  main      INFO:
10-23 11:51:51.447 172.16.161.115:54321  28925  main      INFO: Open H2O Flow in your web browser: http://172.16.161.115:54321
10-23 11:51:51.447 172.16.161.115:54321  28925  main      INFO:
10-23 11:51:51.509 172.16.161.115:54321  28925  main      INFO: Locking cloud to new members, because org.apache.spark.h2o.backends.external.ExternalH2OBackend$$anon$1
10-23 11:51:52.558 172.16.161.115:54321  28925  main      TRACE: H2OContext initialized
ArrayBuffer((0,172.16.161.114,54321))10-23 11:51:57.179 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:51:57.179 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:52:03.730 172.16.161.115:54321  28925  #mCleaner DEBUG: MemGood:   preclean, (K/V:Zero   + POJO:Zero   + FREE:910.5 MB == MEM_MAX:910.5 MB), desiredKV=610.0 MB NO-OOM
10-23 11:52:03.730 172.16.161.115:54321  28925  #mCleaner DEBUG: H(cached:0M, eldest:1571802713363L < +0ms <...{80ms}...< +10240ms < +10367) DESIRED=609M dirtysince=10744 force=false clean2age=5000
10-23 11:52:03.731 172.16.161.115:54321  28925  #mCleaner DEBUG: MemGood:   postclean, (K/V:Zero   + POJO:Zero   + FREE:910.5 MB == MEM_MAX:910.5 MB), desiredKV=610.0 MB NO-OOM
10-23 11:52:03.731 172.16.161.115:54321  28925  #mCleaner DEBUG: Cleaner pass took:  0.001 sec, spilled Zero   in   0 usecH(cached:0M, eldest:1571802713363L < +0ms <...{80ms}...< +10240ms < +10368) diski_o=Zero  , freed=0M, DESIRED=609M
10-23 11:52:07.183 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:52:07.184 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:52:17.184 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:52:17.184 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:52:27.188 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:52:27.188 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:52:37.193 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:52:37.194 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:52:47.194 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:52:47.194 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:52:57.215 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:52:57.215 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:53:07.215 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:53:07.216 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:53:17.216 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:53:17.217 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:53:27.218 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:53:27.218 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:53:37.218 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:53:37.219 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:53:47.291 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:53:47.291 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:53:57.320 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:53:57.321 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:54:07.342 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:54:07.343 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:54:17.374 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:54:17.375 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:54:27.375 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:54:27.375 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:54:37.381 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:54:37.381 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:54:47.386 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:54:47.387 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:54:57.387 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:54:57.387 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:55:07.387 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:55:07.388 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:55:17.412 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:55:17.412 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:55:27.429 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:55:27.430 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:55:37.430 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
10-23 11:55:37.430 172.16.161.115:54321  28925  #ogThread DEBUG: Client: graph3/172.16.161.115:54321(timestamp=-28738, watchdog=false, cloud_name_hash=108209)
10-23 11:55:47.431 172.16.161.115:54321  28925  #ogThread DEBUG: Checking if watchdog client is connected, available clients are:
`
**H2o server logs:**
`
10-23 11:51:49.237 172.16.161.114:54321  2816   FJ-126-7  INFO: Client reported via broadcast message graph3/172.16.161.115:54321 from graph2/172.16.161.114:54321
10-23 11:51:49.529 172.16.161.114:54321  2816   #47:54321 ERRR: Got IO error when sending batch UDP bytes: java.net.ConnectException: 拒绝连接
10-23 11:51:54.533 172.16.161.114:54321  2816   #47:54321 ERRR: Got IO error when sending batch UDP bytes: java.net.ConnectException: 拒绝连接
10-23 11:51:55.222 172.16.161.114:54321  2816   FJ-122-1  INFO: Cancelled remote task#43 class water.TaskGetKey to graph3/172.16.161.115:54321 has been cancelled by remote
`
**My code configuration information**
`
val spark = SparkSession.builder()
      .appName("SW516")
      //.config("spark.ext.h2o.external.read.confirmation.timeout", 300)
      .config("spark.sql.autoBroadcastJoinThreshold", -1)
      .config("spark.locality.wait", 30000)
      .getOrCreate()
    val h2oConf = new H2OConf(spark)
      .setExternalClusterMode()
      .setH2OCluster("172.16.161.114", 54321)
      .useManualClusterStart()
      .setClusterSize(1)
      .setClientVerboseEnabled()
      .setIgnoreSparkPublicDNSEnabled()
      .setClientCheckRetryTimeout(160000) // 160s
      .setCloudTimeout(160000)
      .setH2OClientLogLevel("DEBUG")
      .setUserName("root")
      .set("spark.ext.h2o.cloud.representative", "172.16.161.114:54321")
      .setReplDisabled()
      .setCloudName(cloudName)
    val h2oContext = H2OContext.getOrCreate(spark, h2oConf)
`
comment
created
jakubhava
username_2
@username_0 We have fixed several clouding issues in the code in the upcoming release 3.28.0.3. Can you please give it a try when it's out? We are planning to do a release this week.

Thanks, Kuba
comment
created
jakubhava
username_2
Sparkling Water 3.28.0.3 is released which fixes the clouding issues mentioned above:

Spark 2.4: http://h2o-release.s3.amazonaws.com/sparkling-water/spark-2.4/3.28.0.3-1-2.4/index.html
Spark 2.3: http://h2o-release.s3.amazonaws.com/sparkling-water/spark-2.3/3.28.0.3-1-2.3/index.html
Spark 2.2: http://h2o-release.s3.amazonaws.com/sparkling-water/spark-2.2/3.28.0.3-1-2.2/index.html
Spark 2.1: http://h2o-release.s3.amazonaws.com/sparkling-water/spark-2.1/3.28.0.3-1-2.1/index.html

If you bump into any new issues, please create new or feel free to reopen this issue.
comment
closed
jakubhava
username_2
issue