Patent ID: 11914496
Assignee: DATA CULPA, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 7:
8. A method of operating a data pipeline monitoring system configured to maintain data consistency in a data pipeline, the method comprising:
data monitoring circuitry monitoring operations of the data pipeline wherein the data pipeline receives input data sets, processes the input data sets, and responsively generates and transfers output data sets;
the data monitoring circuitry receiving the output data sets, processing the output data sets to identify changes between the output data sets, utilizing a Sigmoid function to calculate weighted sums of data values, data records, and data schemas for the output data sets, generating a consistency score for the output data sets based on the changes and the weighted sums, and determining when the consistency score exceeds a threshold value, wherein the consistency score indicates a similarity level between the output data sets; and
when the consistency score exceeds the threshold value, the data monitoring circuitry generating and transferring an alert that indicates ones of the output data sets that exceeded the threshold value.