Bug: Spark MasterWebUI and WorkerWebUI fail to start when NSSDB used as keystore, getting java.security.KeyStoreException: PKCS11 not found.
Status: Open | Priority: Major | Resolution: 
Created: 21/Feb/23 13:13
Affects Version/s: 3.3.0

When we are running Spark by setting up below SSL configurations, Spark masterwebui and workerwebui is fail to start.
{quote}
        "spark.ssl.enabled":"true",
        "spark.ssl.keyStore":"/opt/ibm/jdk/conf/security/nss.fips.cfg",
        "spark.ssl.keyStorePassword":"<keystore passwd>",
        "spark.ssl.keyStoreType":"PKCS11"
{quote}

*Errors :*

{quote}23/02/21 12:29:43 INFO Master: Running Spark version 3.3.1
23/02/21 12:29:43 ERROR MasterWebUI: Failed to bind MasterWebUI
java.security.KeyStoreException: PKCS11 not found
	at java.base/java.security.KeyStore.getInstance(KeyStore.java:878)
	at org.sparkproject.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:46)
	at org.sparkproject.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1203)
	at org.sparkproject.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:322)
	at org.sparkproject.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:244)
	at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.sparkproject.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.sparkproject.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.sparkproject.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:97)
	at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.sparkproject.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.sparkproject.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.sparkproject.jetty.server.AbstractConnector.doStart(AbstractConnector.java:323)
	at org.sparkproject.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
	at org.sparkproject.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
	at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.apache.spark.ui.JettyUtils$.newConnector$1(JettyUtils.scala:303)
	at org.apache.spark.ui.JettyUtils$.sslConnect$1(JettyUtils.scala:322)
	at org.apache.spark.ui.JettyUtils$.$anonfun$startJettyServer$4(JettyUtils.scala:326)
	at org.apache.spark.ui.JettyUtils$.$anonfun$startJettyServer$4$adapted(JettyUtils.scala:326)
	at org.apache.spark.util.Utils$.$anonfun$startServiceOnPort$2(Utils.scala:2401)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
	at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2393)
	at org.apache.spark.ui.JettyUtils$.$anonfun$startJettyServer$2(JettyUtils.scala:326)
	at org.apache.spark.ui.JettyUtils$.$anonfun$startJettyServer$2$adapted(JettyUtils.scala:315)
	at scala.Option.map(Option.scala:230)
	at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:315)
	at org.apache.spark.ui.WebUI.initServer(WebUI.scala:144)
	at org.apache.spark.ui.WebUI.bind(WebUI.scala:153)
	at org.apache.spark.deploy.master.Master.onStart(Master.scala:138)
	at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:120)
	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
	at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
	at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
	at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:839)
Caused by: java.security.NoSuchAlgorithmException: PKCS11 KeyStore not available
	at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
	at java.base/java.security.Security.getImpl(Security.java:719)
	at java.base/java.security.KeyStore.getInstance(KeyStore.java:875)
	... 37 more
{quote}

content of nss fips config file.

{quote}name = NSS-FIPS
nssLibraryDirectory = /usr/lib64
nssSecmodDirectory = /etc/pki/nssdb
nssDbMode = readOnly
nssModule = fips

attributes(*,CKO_SECRET_KEY,CKK_GENERIC_SECRET)={ CKA_SIGN=true }{quote}