Bug: Fix Flaky HiveThriftHttpServerSuite
Status: In Progress | Priority: Major | Resolution: 
Created: 03/Nov/20 18:55
Affects Version/s: 3.1.0

After launching successfully `HiveThriftServer2 started successfully`, the test fails due to 500 error.

The failure rate is over 50%. (This is an example of the test case `JDBC query execution` in that suite)
 !failure_rate.png|width=508,height=321!
 - [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/1516/testReport/]
{code:java}
09:58:03.853 pool-1-thread-1 INFO HiveThriftHttpServerSuite: Trying to start HiveThriftServer2: port=14541, mode=http, attempt=0
09:58:06.492 pool-1-thread-1 INFO HiveThriftHttpServerSuite: COMMAND: WrappedArray(../../sbin/start-thriftserver.sh, --master, local, --hiveconf, javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=/home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-hive-2.3/target/tmp/spark-6f4abc35-f09c-46e6-b6eb-8a310d557e28;create=true, --hiveconf, hive.metastore.warehouse.dir=/home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-hive-2.3/target/tmp/spark-2329343a-3ad4-4bfd-943f-6b46984848b8, --hiveconf, hive.server2.thrift.bind.host=localhost, --hiveconf, hive.server2.transport.mode=http, --hiveconf, hive.server2.logging.operation.log.location=/home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-hive-2.3/target/tmp/spark-77f3b359-1553-40e3-9d75-35c46d2d4d46, --hiveconf, hive.exec.local.scratchdir=/home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-hive-2.3/target/tmp/spark-8923e61f-36da-4930-b035-6eb3712d41ab, --hiveconf, hive.server2.thrift.http.port=14541, --driver-class-path, /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-hive-2.3/target/tmp/spark-d54a7073-2f02-4331-a84d-bbb3b50a47ac, --driver-java-options, -Dlog4j.debug, --conf, spark.ui.enabled=false)
09:58:06.492 pool-1-thread-1 INFO HiveThriftHttpServerSuite: OUTPUT: starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2, logging to /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-hive-2.3/logs/spark-jenkins-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-amp-jenkins-worker-06.out

09:58:38.688 pool-1-thread-1 INFO HiveThriftHttpServerSuite: HiveThriftServer2 started successfully
09:58:38.689 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite INFO HiveThriftHttpServerSuite:

===== TEST OUTPUT FOR o.a.s.sql.hive.thriftserver.HiveThriftHttpServerSuite: 'JDBC query execution' =====

09:58:38.691 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite INFO Utils: Supplied authorities: localhost:14541
09:58:38.691 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite WARN Utils: ***** JDBC param deprecation *****
09:58:38.691 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite WARN Utils: The use of hive.server2.transport.mode is deprecated.
09:58:38.691 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite WARN Utils: Please use transportMode like so: jdbc:hive2://<host>:<port>/dbName;transportMode=<transport_mode_value>
09:58:38.691 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite WARN Utils: ***** JDBC param deprecation *****
09:58:38.691 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite WARN Utils: The use of hive.server2.thrift.http.path is deprecated.
09:58:38.691 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite WARN Utils: Please use httpPath like so: jdbc:hive2://<host>:<port>/dbName;httpPath=<http_path_value>
09:58:38.692 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite INFO Utils: Resolved authority: localhost:14541
09:58:38.818 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG RequestAddCookies: CookieSpec selected: default
09:58:38.830 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG RequestAuthCache: Auth cache not set in the context
09:58:38.832 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:14541][total available: 0; route allocated: 0 of 2; total allocated: 0 of 20]
09:58:38.846 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:14541][total available: 0; route allocated: 1 of 2; total allocated: 1 of 20]
09:58:38.847 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG MainClientExec: Opening connection {}->http://localhost:14541
09:58:38.850 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:14541
09:58:38.851 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:39668<->127.0.0.1:14541
09:58:38.851 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG MainClientExec: Executing request POST /cliservice HTTP/1.1
09:58:38.852 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
09:58:38.854 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG headers: http-outgoing-0 >> POST /cliservice HTTP/1.1
09:58:38.854 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG headers: http-outgoing-0 >> Content-Type: application/x-thrift
09:58:38.854 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG headers: http-outgoing-0 >> Accept: application/x-thrift
09:58:38.854 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG headers: http-outgoing-0 >> User-Agent: Java/THttpClient/HC
09:58:38.854 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG headers: http-outgoing-0 >> Authorization: Basic amVua2luczphbm9ueW1vdXM=
09:58:38.854 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG headers: http-outgoing-0 >> Content-Length: 254
09:58:38.854 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG headers: http-outgoing-0 >> Host: localhost:14541
09:58:38.854 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
09:58:38.854 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG headers: http-outgoing-0 >> Accept-Encoding: gzip,deflate
09:58:38.854 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG headers: http-outgoing-0 >> X-XSRF-HEADER: true
09:58:38.855 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 >> "POST /cliservice HTTP/1.1[\r][\n]"
09:58:38.855 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 >> "Content-Type: application/x-thrift[\r][\n]"
09:58:38.855 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 >> "Accept: application/x-thrift[\r][\n]"
09:58:38.855 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 >> "User-Agent: Java/THttpClient/HC[\r][\n]"
09:58:38.855 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 >> "Authorization: Basic amVua2luczphbm9ueW1vdXM=[\r][\n]"
09:58:38.855 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 >> "Content-Length: 254[\r][\n]"
09:58:38.855 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 >> "Host: localhost:14541[\r][\n]"
09:58:38.855 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
09:58:38.855 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
09:58:38.855 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 >> "X-XSRF-HEADER: true[\r][\n]"
09:58:38.855 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 >> "[\r][\n]"
09:58:38.855 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 >> "[0x80][0x1][0x0][0x1][0x0][0x0][0x0][0xb]OpenSession[0x0][0x0][0x0][0x1][0xc][0x0][0x1][0x8][0x0][0x1][0x0][0x0][0x0][0x9][\r][0x0][0x4][0xb][0xb][0x0][0x0][0x0][0x6][0x0][0x0][0x0][0xe]set:hiveconf:a[0x0][0x0][0x0][0x6]avalue[0x0][0x0][0x0]=set:hiveconf:hive.server2.thrift.resultset.default.fetch.size[0x0][0x0][0x0][0x4]1000[0x0][0x0][0x0][\r]set:hivevar:c[0x0][0x0][0x0][0x6]cvalue[0x0][0x0][0x0][0xe]set:hiveconf:b[0x0][0x0][0x0][0x6]bvalue[0x0][0x0][0x0][\r]set:hivevar:d[0x0][0x0][0x0][0x6]dvalue[0x0][0x0][0x0][0xc]use:database[0x0][0x0][0x0][0x7]default[0x0][0x0]"
09:58:39.020 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 << "HTTP/1.1 500 Server Error[\r][\n]"
09:58:39.020 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 << "Cache-Control: must-revalidate,no-cache,no-store[\r][\n]"
09:58:39.020 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 << "Connection: close[\r][\n]"
09:58:39.020 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 << "Server: Jetty(9.4.28.v20200408)[\r][\n]"
09:58:39.020 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG wire: http-outgoing-0 << "[\r][\n]"
09:58:39.023 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG headers: http-outgoing-0 << HTTP/1.1 500 Server Error
09:58:39.023 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG headers: http-outgoing-0 << Cache-Control: must-revalidate,no-cache,no-store
09:58:39.023 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG headers: http-outgoing-0 << Connection: close
09:58:39.023 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG headers: http-outgoing-0 << Server: Jetty(9.4.28.v20200408)
09:58:39.028 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: Close connection
09:58:39.028 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG MainClientExec: Connection discarded
09:58:39.028 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:14541][total available: 0; route allocated: 0 of 2; total allocated: 0 of 20]
09:58:39.028 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite DEBUG MainClientExec: Cancelling request execution
09:58:39.029 pool-1-thread-1-ScalaTest-running-HiveThriftHttpServerSuite ERROR HiveConnection: Error opening session
org.apache.thrift.transport.TTransportException: HTTP Response code: 500
        at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:262)
        at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:316)
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
        at org.apache.hive.service.rpc.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:162)
        at org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:154)
        at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:680)
        at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:200)
        at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftJdbcTest.$anonfun$withMultipleConnectionJdbcStatement$1(HiveThriftServer2Suites.scala:1141)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
        at scala.collection.TraversableLike.map(TraversableLike.scala:238)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftJdbcTest.withMultipleConnectionJdbcStatement(HiveThriftServer2Suites.scala:1141)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftJdbcTest.withJdbcStatement(HiveThriftServer2Suites.scala:1177)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftHttpServerSuite.$anonfun$new$119(HiveThriftServer2Suites.scala:1086)
{code}