A method performed by one or more computing devices, the method comprising: receiving a plurality of source data records from one or more source computers, wherein one or more first source data records of said plurality of source data records are associated with a first source transaction and one or more second source data records of said plurality of source data records are associated with a second source transaction; generating a first derived transaction comprising one or more first derived records generated based on the plurality of source data records; generating a first transaction mapping between the first derived transaction and the first source transaction; generating a second transaction mapping between the first derived transaction and the second source transaction, the method being characterized by: determining that the first derived transaction has ended, and in response, committing the first derived transaction including the one or more first derived records to a persistent storage; receiving a first ending punctuation associated with the first source transaction, wherein the ending punctuation includes data that indicates an end of the first source transaction, and in response, committing the first source transaction including the one or more first source data records associated with the first source transaction to the persistent storage; after committing the first derived transaction, determining that the second source transaction failed, and in response: determining that the first derived transaction should be regenerated based on the second transaction mapping, and in response, regenerating the first derived transaction by: generating one or more second derived records that are based on the one or more first source data records but not the one or more second source data records; committing the one or more second derived records to the persistent storage. receiving a plurality of source data records from one or more source computers, wherein one or more first source data records of said plurality of source data records are associated with a first source transaction and one or more second source data records of said plurality of source data records are associated with a second source transaction; generating a first derived transaction comprising one or more first derived records generated based on the plurality of source data records; generating a first transaction mapping between the first derived transaction and the first source transaction; generating a second transaction mapping between the first derived transaction and the second source transaction, the method being characterized by: determining that the first derived transaction has ended, and in response, committing the first derived transaction including the one or more first derived records to a persistent storage; receiving a first ending punctuation associated with the first source transaction, wherein the ending punctuation includes data that indicates an end of the first source transaction, and in response, committing the first source transaction including the one or more first source data records associated with the first source transaction to the persistent storage; after committing the first derived transaction, determining that the second source transaction failed, and in response: determining that the first derived transaction should be regenerated based on the second transaction mapping, and in response, regenerating the first derived transaction by: generating one or more second derived records that are based on the one or more first source data records but not the one or more second source data records; committing the one or more second derived records to the persistent storage. determining that the first derived transaction has ended, and in response, committing the first derived transaction including the one or more first derived records to a persistent storage; receiving a first ending punctuation associated with the first source transaction, wherein the ending punctuation includes data that indicates an end of the first source transaction, and in response, committing the first source transaction including the one or more first source data records associated with the first source transaction to the persistent storage; after committing the first derived transaction, determining that the second source transaction failed, and in response: determining that the first derived transaction should be regenerated based on the second transaction mapping, and in response, regenerating the first derived transaction by: generating one or more second derived records that are based on the one or more first source data records but not the one or more second source data records; committing the one or more second derived records to the persistent storage. generating one or more second derived records that are based on the one or more first source data records but not the one or more second source data records; committing the one or more second derived records to the persistent storage.