Bug: Stream-stream join with one watermark and one time-interval join leads to improper state cleanup
Status: Open | Priority: Major | Resolution: 
Created: 10/Dec/23 07:14
Affects Version/s: 3.5.0

h2. Summary

State from stream-stream joins is not cleaned up properly in the following condition:
 * The user has specified one watermark. Without loss of generality, say that this is the left side.
 * The user is doing a time-interval join, with an inequality of the form `L > R + k`, where `k` can be either positive or negative (doesn't matter)

Under these conditions, state _should_ be removed from the left-hand side using the right-hand side watermark, but it's not. However, Spark currently does not clean up any state from the left-hand side. You can find a repro [here|https://github.com/neilramaswamy/nr-spark/pull/1/files#diff-a271c249e6989ebaad63a491910597cd336049516e5b8e9a80f2bbd7368f458dR260].
h2. Brief aside on state removal

Using the conditions in the bullet list above, here's how we know we can remove state from the left.

If we know the minimum right event-time we'll receive in the future is `r` (this is the right watermark), then the only future records on the left that could join have event-time greater than `r + k`. As a result, we can remove every left record with event-time _less_ than or equal to that value, `r + k`.

Taking this logic without loss of generality (i.e. knowing the minimum left event-time, and having a time-interval condition of the form `R > L + c`), we can remove state from the right. That thus yields state removal from both sides.
h2. Probable cause of the issue

I believe the reason that we do not clean up state correctly is because of a bug in the state watermark generation predicate. In particular, [this line|https://github.com/apache/spark/blob/bac3492980a3e793065a9e9d511ddf0fb66357b3/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinHelper.scala#L205] should be checking whether the watermark exists on the _other_ side. Since we want to drop state using `oneSideInputAttributes`, we need to look for the watermark in `otherSideInputAttributes`. Concretely, if we're trying to drop state for the left, we need to check if the watermark exists on the right. Because our check is wrong here, `isWatermarkDefinedOnJoinKey` becomes false in this case, so [here|https://github.com/apache/spark/blob/bac3492980a3e793065a9e9d511ddf0fb66357b3/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinHelper.scala#L208], we short-circuit and never actually end up applying the previously computed `stateValueWatermark`.
h2. Why we didn't notice this before

Our streaming join test suite actually watermarks every stream in all examples. For this reason, any side on which we wanted to drop state had its own watermark, so [this line|https://github.com/apache/spark/blob/bac3492980a3e793065a9e9d511ddf0fb66357b3/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinHelper.scala#L208] would have always returned true.

Additionally, our programming guide has an example with watermarks defined on both sides, so most users probably didn't try to run a stream-stream join with just one watermark.
h2. Proposed (proof-of-concept) change

You can find a POC fix [here|https://github.com/neilramaswamy/nr-spark/pull/1/files], where you can also find test-cases that fail against our master branch, but succeed with the changes.