Bug: Spark connect client failing with java.lang.NoClassDefFoundError
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Sep/23 14:29
Affects Version/s: 3.5.0

java 1.8, sbt 1.9, scala 2.12

 

I have a very simple repo with the following dependency in `build.sbt`

```

{{libraryDependencies ++= Seq("org.apache.spark" %% "spark-connect-client-jvm" % "3.5.0")}}

```

A simple application

```

{{object Main extends App {}}
{{   val s = SparkSession.builder().remote("sc://localhost").getOrCreate()}}
{{   s.read.json("/tmp/input.json").repartition(10).show(false)}}
{{}}}

```

But when I run it, I get the following error

 

```

{{Exception in thread "main" java.lang.NoClassDefFoundError: org/sparkproject/connect/client/com/google/common/cache/CacheLoader}}
{{    at Main$.delayedEndpoint$Main$1(Main.scala:4)}}
{{    at Main$delayedInit$body.apply(Main.scala:3)}}
{{    at scala.Function0.apply$mcV$sp(Function0.scala:39)}}
{{    at scala.Function0.apply$mcV$sp$(Function0.scala:39)}}
{{    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)}}
{{    at scala.App.$anonfun$main$1$adapted(App.scala:80)}}
{{    at scala.collection.immutable.List.foreach(List.scala:431)}}
{{    at scala.App.main(App.scala:80)}}
{{    at scala.App.main$(App.scala:78)}}
{{    at Main$.main(Main.scala:3)}}
{{    at Main.main(Main.scala)}}
{{Caused by: java.lang.ClassNotFoundException: org.sparkproject.connect.client.com.google.common.cache.CacheLoader}}
{{    at java.net.URLClassLoader.findClass(URLClassLoader.java:387)}}
{{    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)}}
{{    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)}}
{{    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)}}
{{    ... 11 more}}

```

I know the connect does a bunch of shading during assembly so it could be related to that. This application is not started via spark-submit or anything. It's not run neither under a `SPARK_HOME` ( I guess that's the whole point of connect client )

 

EDIT

Not sure if it's the right mitigation but explicitly adding guava worked but now I am in the 2nd territory of error

{{Sep 21, 2023 8:21:59 PM org.sparkproject.connect.client.io.grpc.NameResolverRegistry getDefaultRegistry}}
{{WARNING: No NameResolverProviders found via ServiceLoader, including for DNS. This is probably due to a broken build. If using ProGuard, check your configuration}}
{{Exception in thread "main" org.sparkproject.connect.client.com.google.common.util.concurrent.UncheckedExecutionException: org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry$ProviderNotFoundException: No functional channel service provider found. Try adding a dependency on the grpc-okhttp, grpc-netty, or grpc-netty-shaded artifact}}
{{    at org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2085)}}
{{    at org.sparkproject.connect.client.com.google.common.cache.LocalCache.get(LocalCache.java:4011)}}
{{    at org.sparkproject.connect.client.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4034)}}
{{    at org.sparkproject.connect.client.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5010)}}
{{    at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:945)}}
{{    at scala.Option.getOrElse(Option.scala:189)}}
{{    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:945)}}
{{    at Main$.delayedEndpoint$Main$1(Main.scala:4)}}
{{    at Main$delayedInit$body.apply(Main.scala:3)}}
{{    at scala.Function0.apply$mcV$sp(Function0.scala:39)}}
{{    at scala.Function0.apply$mcV$sp$(Function0.scala:39)}}
{{    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)}}
{{    at scala.App.$anonfun$main$1$adapted(App.scala:80)}}
{{    at scala.collection.immutable.List.foreach(List.scala:431)}}
{{    at scala.App.main(App.scala:80)}}
{{    at scala.App.main$(App.scala:78)}}
{{    at Main$.main(Main.scala:3)}}
{{    at Main.main(Main.scala)}}
{{Caused by: org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry$ProviderNotFoundException: No functional channel service provider found. Try adding a dependency on the grpc-okhttp, grpc-netty, or grpc-netty-shaded artifact}}
{{    at org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry.newChannelBuilder(ManagedChannelRegistry.java:179)}}
{{    at org.sparkproject.connect.client.io.grpc.ManagedChannelRegistry.newChannelBuilder(ManagedChannelRegistry.java:155)}}
{{    at org.sparkproject.connect.client.io.grpc.Grpc.newChannelBuilder(Grpc.java:101)}}
{{    at org.sparkproject.connect.client.io.grpc.Grpc.newChannelBuilderForAddress(Grpc.java:111)}}
{{    at org.apache.spark.sql.connect.client.SparkConnectClient$Configuration.createChannel(SparkConnectClient.scala:633)}}
{{    at org.apache.spark.sql.connect.client.SparkConnectClient$Configuration.toSparkConnectClient(SparkConnectClient.scala:645)}}
{{    at org.apache.spark.sql.SparkSession$.create(SparkSession.scala:760)}}
{{    at org.apache.spark.sql.SparkSession$$anon$1.load(SparkSession.scala:736)}}
{{    at org.apache.spark.sql.SparkSession$$anon$1.load(SparkSession.scala:735)}}
{{    at org.sparkproject.connect.client.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3570)}}
{{    at org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2312)}}
{{    at org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189)}}
{{    at org.sparkproject.connect.client.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079)}}
{{    ... 17 more}}

 

The fix as per the error suggestion isn't working :/ 

I followed the doc exactly as described. Can somebody help