Bug: Avoid testing the `streaming-kinesis-asl` module in the daily tests of branch-3.x.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Dec/23 08:36
Affects Version/s: 4.0.0

After the merge of https://github.com/apache/spark/pull/43736, the master branch began testing the `streaming-kinesis-asl` module. 

At the same time, because the daily test will reuse `build_and_test.yml`, the daily test of branch-3.x also began testing `streaming-kinesis-asl`. 

However, in branch-3.x, the env `ENABLE_KINESIS_TESTS` is hard-coded as 1 in `dev/sparktestsupport/modules.py`:

https://github.com/apache/spark/blob/1321b4e64deaa1e58bf297c25b72319083056568/dev/sparktestsupport/modules.py#L332-L346

which leads to the failure of the daily test of branch-3.x:

- branch-3.3: https://github.com/apache/spark/actions/runs/7111246311
- branch-3.4: https://github.com/apache/spark/actions/runs/7098435892
- branch-3.5: https://github.com/apache/spark/actions/runs/7099811235

```
[info] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite *** ABORTED *** (1 second, 14 milliseconds)
[info]   java.lang.Exception: Kinesis tests enabled using environment variable ENABLE_KINESIS_TESTS
[info] but could not find AWS credentials. Please follow instructions in AWS documentation
[info] to set the credentials in your system such that the DefaultAWSCredentialsProviderChain
[info] can find the credentials.
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils$.getAWSCredentials(KinesisTestUtils.scala:258)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61)
[info]   at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info]   at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42)
[info]   at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
[info]   at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info]   at java.lang.Thread.run(Thread.java:750)
[info] Test run org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite started
[info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite.testJavaKinesisDStreamBuilderOldApi started
[info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite.testJavaKinesisDStreamBuilder started
[info] Test run org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite finished: 0 failed, 0 ignored, 2 total, 0.244s
[info] ScalaTest
[info] Run completed in 8 seconds, 542 milliseconds.
[info] Total number of tests run: 31
[info] Suites: completed 4, aborted 4
[info] Tests: succeeded 31, failed 0, canceled 0, ignored 0, pending 0
[info] *** 4 SUITES ABORTED ***
[error] Error: Total 37, Failed 0, Errors 4, Passed 33
[error] Error during tests:
[error]   org.apache.spark.streaming.kinesis.WithoutAggregationKinesisBackedBlockRDDSuite
[error]   org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite
[error]   org.apache.spark.streaming.kinesis.WithAggregationKinesisStreamSuite
[error]   org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite
[error] (streaming-kinesis-asl / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 13 s, completed Dec 5, 2023 12:03:53 PM
```