Repository: feldera/feldera

Issue #3558: [RFC] State reuse across different versions of a pipeline (aka backfill avoidance)
Created: 2025-02-19T01:43:08Z
TODOs:


 DBSP support for internal backfill (#3865 )


 adapters support for internal backfill (#3909)


 #3838


 #3839


 #3841


 #3842


 #3843


 #3858



This RFC describes an algorithm that bootstraps a modified version of a DBSP circuit while reusing state accumulated by the previous version of the circuit when possible.
Replaying Stored Streams
In this algorithm we use the following technique to replay streams whose contents is stored in an integrator.  An integrator in DBSP looks like this:
             │              
             │              
             │              
   ┌─────────┤              
   │         │           I  
   │    ┌──────────────────┐
   │    │    ▼             │
   │    │  ┌───┐     ┌───┐ │
   │    │  │ + │◄──┬─┤Z-1│ │
   │    │  └─┬─┘   │ └───┘ │
   │    │    │     │   ▲   │
   │    │    │     │   │   │
   │    │    ├─────┼───┘   │
   │    │    │     │       │
   │    └──────────┼───────┘
   │         │     │        
s1 │       s2│  s3 │        
   │         │     │
   ▼         ▼     ▼

Here s1 is a stream of changes to some Z-set; s2 is the integral of the stream, and s3 is the delayed integral, i.e., the value of the integral from the previous step.  Or, treating the integral as a black box:
             │    
  ┌──────────┤    
  │          │    
  │          ▼    
  │      ┌───────┐
  │      │       │
  │      │   I   │
  │      │       │
  │      └┬─────┬┘
  │       │     │ 
  │       │     │ 
  │       │     │ 
s1│     s2│   s3│ 
  ▼       ▼     ▼ 

If we discover that one or more of the three streams needs to be replayed in order to populate new parts of the circuit, use the following procedure:

Remove the saved state from the integrator, replacing it with an empty Z-set.
Push the saved state to the s1 stream. In principle, we could push the entire state in one step, but since we currently don't support giant steps well, the other option is to break it up into chunks, and feed one chunk per step. The recovery procedure described below guarantees that the circuit will converge to the correct final state.

             │ replay  ┌───────────┐
  ┌──────────┤◄────────│saved state│
  │          │         └───────────┘
  │          ▼                      
  │      ┌───────┐                  
  │      │       │                  
  │      │   I   │                  
  │      │       │                  
  │      └┬─────┬┘                  
  │       │     │                   
  │       │     │                   
  │       │     │                   
s1│     s2│   s3│                   
  ▼       ▼     ▼                   

The algorithm
Step 1. Identify and label common parts of the circuit.
At this step we identify and label parts of the circuit that haven't changed between the old and new versions. We will need compiler support here.
The algorithm labels each operator in the graph as either OLD or NEW. The OLD label indicates that this operator and all its transitive predecessors existed in the old version of the circuit, and therefore any state the operator may have accumulated during the previous run can be reused.

Label old input nodes.  An input node is a SQL table with a set of connectors attached to it. An input node is labeled as OLD if


an identical table declaration with an identical list of connectors existed in the old version of the circuit, and
the table is materialized (see the discussion below), and
all connectors attached to the table are fault tolerant, i.e., can be configured to continue ingesting from exactly the point where the old instance of the connector stopped.



Identify the maximal connected fragment of the new circuit, reachable from OLD input nodes, identical to a fragment of the old circuit. Label all operators in this fragment as OLD.


Label all remaining operators as NEW.


Once all operators have been labeled, we can initialize all stateful old operators with the state accumulated by the previous version of the pipeline, and discard any other stored state. Alternatively, we can keep the old state until the recovery process completes successfully.
Step 2. Prepare the circuit for the catch-up phase
The catch-up phase is a new execution mode of a DBSP circuit where new operators are initialized using the preserved state of the old circuit.
During the catch-up phase, an operator can be doing one of three things:

Remain idle. Idle operators are not scheduled during the catch-up phase.
Run in normal mode. These operators process inputs and produce outputs just like during normal operation of the circuit.
Replay mode. A subset of integrator operators run in replay mode, where the contents of the integral is re-ingested into the circuit, as explained above.

During step 2, we label each operator in the circuit accordingly as IDLE, RUN, or REPLAY using the following procedure:


Identify all OLD integrals in the circuit and label the input and output streams of each such integral as STORED streams. A stored stream can be replayed during the catch-up phase from the contents of the integral.


Label all NEW operators as RUN.


Consider all input streams of all operators labeled RUN, which have not been encountered yet.

If the stream is labeled as STORED, label its associated integrator operator with REPLAY.
Otherwise, label the origin of the stream as RUN.

Repeat Step 3 until reaching a fixed point.


Label all unlabeled operators as IDLE.


NOTE: This algorithm labels all OLD input operators as IDLE. This is due to the requirement that all OLD tables must be materialized, which guarantees that all OLD inputs are stored. Therefore, none of the input connectors will be active during the catch-up phase.
Step 3. Catch-up
During the catch-up phase, we run the circuit, scheduling operators according to the three labels computed during the previous step:

IDLE operators are not evaluated. The scheduler treats them as if they weren't part of the circuit.
RUN operators are evaluated exactly as during normal operation.
REPLAY operators are evaluated in replay mode: the state of the integral is replaced with the empty Z-set, and injected into the input stream of the integral as described above.

The circuit runs in this mode until the contents of the integrals has been completely injected into the circuit.
Step 4. Normal operation.
After the catch-up phase completes the circuit switches to normal operation.

Duplicate issue #3977: Support checkpointing of input tables only
Created: 2025-05-06T18:02:37Z
Currently suspending a pipeline checkpoints the entire state. For S3 storage, the pipeline will become instantly runnable, but can take a while to warm up the caches and will probably be quite slow until it does.
We wish to add support for checkpoint input tables only. In this case, the pipeline cannot produce new outputs until it has read the entire input snapthot from S3 and bootstrapped its internal state from that.
In the latter case, the tables that are not checkpointed would use local storage (presumably an SSD).
To be specified, among other elements:

Configuration (we'll have to figure out where this fits and how to describe it).
Distinguishing the two kinds of state and where they should go (need help from @ryzhyk ).

Closing comment (2025-05-06T18:14:56Z): This is also related to #3558 and #3925.  Internally this will be implemented by bootstrapping intermediate state from the input checkpoint. Huge-step support will allow us to do this more efficiently.