Bug: Cannot run spark history server with shaded hadoop jar
Status: Open | Priority: Major | Resolution: 
Created: 29/Oct/22 04:06
Affects Version/s: 3.2.2

Since SPARK-33212, Spark uses shaded client jars from Hadoop 3.x+.
If you try to start Spark History Server with shaded client jars and enable security using org.apache.hadoop.security.authentication.server.AuthenticationFilter, you will meet following exception.

{code}
# spark-env.sh
export SPARK_HISTORY_OPTS='-Dspark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter -Dspark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=kerberos,kerberos.principal=HTTP/some.example.com@EXAMPLE.COM,kerberos.keytab=/etc/security/keytabs/spnego.service.keytab"'
{code}


{code}
# spark history server's out file
22/10/27 15:29:48 INFO AbstractConnector: Started ServerConnector@5ca1f591{HTTP/1.1, (http/1.1)}{0.0.0.0:18081}
22/10/27 15:29:48 INFO Utils: Successfully started service 'HistoryServerUI' on port 18081.
22/10/27 15:29:48 INFO ServerInfo: Adding filter to /: org.apache.hadoop.security.authentication.server.AuthenticationFilter
22/10/27 15:29:48 ERROR HistoryServer: Failed to bind HistoryServer
java.lang.IllegalStateException: class org.apache.hadoop.security.authentication.server.AuthenticationFilter is not a javax.servlet.Filter
        at org.sparkproject.jetty.servlet.FilterHolder.doStart(FilterHolder.java:103)
        at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.sparkproject.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:730)
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
        at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
        at org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:755)
        at org.sparkproject.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
        at org.sparkproject.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:910)
        at org.sparkproject.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
        at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:491)
        at org.apache.spark.ui.WebUI.$anonfun$bind$3(WebUI.scala:148)
        at org.apache.spark.ui.WebUI.$anonfun$bind$3$adapted(WebUI.scala:148)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at org.apache.spark.ui.WebUI.bind(WebUI.scala:148)
        at org.apache.spark.deploy.history.HistoryServer.bind(HistoryServer.scala:164)
        at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:310)
        at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
{code}


I think "AuthenticationFilter" in the shaded jar imports "org.apache.hadoop.shaded.javax.servlet.Filter", not "javax.servlet.Filter".

{code}
❯ grep -r org.apache.hadoop.shaded.javax.servlet.Filter *
Binary file hadoop-client-runtime-3.3.1.jar matches
{code}

It causes the exception I mentioned.

I'm not sure what is the best answer.
Workaround is not to use spark with pre-built for Apache Hadoop, specify `HADOOP_HOME` or `SPARK_DIST_CLASSPATH` in spark-env.sh for Spark History Server.

May be the possible options are:
- Not to shade "javax.servlet.Filter" at hadoop shaded jar
- Or, shade "javax.servlet.Filter" also at jetty.