Bug: java.lang.NoClassDefFoundError: javax/servlet/Servlet incompatibilities upgrading springboot-dependencies 2.7 to 3.* (package javax.* -> jakarta.*)
Status: Open | Priority: Major | Resolution: 
Created: 12/Nov/23 18:10
Affects Version/s: 3.2.4

updating springboot-dependencies 2.7 to 3.1 breaks spark with springboot compatibilities.


{noformat}
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>2.7.14</version> <!-- OK 2.7.* works fine, but  3.* ERROR --> 
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
{noformat}

Internally, everything compile ok, but fail at runtime while creating the SparkSession, with SparkUI:


{noformat}
Caused by: java.lang.NoClassDefFoundError: javax/servlet/Servlet
	at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:239) ~[spark-core_2.13-3.5.0.jar:3.5.0]
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:503) ~[spark-core_2.13-3.5.0.jar:3.5.0]
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888) ~[spark-core_2.13-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1099) ~[spark-sql_2.13-3.5.0.jar:3.5.0]
	at scala.Option.getOrElse(Option.scala:201) ~[scala-library-2.13.8.jar:na]
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1093) ~[spark-sql_2.13-3.5.0.jar:3.5.0]
....

Caused by: java.lang.ClassNotFoundException: javax.servlet.Servlet
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[na:na]
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[na:na]
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) ~[na:na]
	... 49 common frames omitted

{noformat}

Root cause: major change with springboot 3: 'javax.servlet.Servlet' class is NO more defined in package 'javax.servlet.', but migrated to 'jakarta.servlet.'

The compiled dependency from spark-core to javax-servlet-api is replaced by maven dependencyManagement


It "might be" possible to get rid of maven dependencyManagement or to override several artifact versions, but it is NOT PRACTICAL.

Here is a version that work, much more complex  with springboot3 than springboot2.

It requires few of forced versions, and exclusions in maven.

    
{noformat}
<properties>
        <spark.version>3.5.0</spark.version>
        <scala.version>2.13</scala.version>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>3.1.2</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>org.glassfish.jersey.containers</groupId>
                <artifactId>jersey-container-servlet-core</artifactId>
                <version>2.41</version>
            </dependency>
            <dependency>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
                <version>2.0.9</version>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.logging.log4j</groupId>
                    <artifactId>log4j-slf4j2-impl</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>jakarta.servlet</groupId>
            <artifactId>jakarta.servlet-api</artifactId>
            <version>4.0.4</version>
        </dependency>

    </dependencies>
{noformat}


there are several examples showing errors and fixed for servlet, log4j, glassfish incompatibilities here as jira attachment, or directly from github: https://github.com/Arnaud-Nauwynck/test-snippets/tree/master/test-springboot-spark



A possible long term solution is to migrate to servlet in package "jakarta.servlet." instead of "javax.servlet." as springboot already did, so there would "not (?)" be incompatibilities between very old and much more recent jars.

Any roadmap for this?