Patent ID: 11914496
Assignee: DATA CULPA, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A data pipeline monitoring system configured to maintain data consistency in a data pipeline, the data pipeline monitoring system comprising:
data monitoring circuitry configured to monitor operations of the data pipeline wherein the data pipeline receives input data sets, processes the input data sets, and responsively generates and transfers output data sets;
the data monitoring circuitry configured to receive the output data sets, process the output data sets to identify changes between the output data sets, utilize a Sigmoid function to calculate weighted sums of data values, data records, and data schemas for the output data sets, generate a consistency score for the output data sets based on the changes and the weighted sums, and determine when the consistency score exceeds a threshold value, wherein the consistency score indicates a similarity level between the output data sets; and
when the consistency score exceeds the threshold value, the data monitoring circuitry configured to generate and transfer an alert that indicates ones of the output data sets that exceeded the threshold value.