Project: talenddataquality
Issue TDQ-12213: filter data in the tMatchPairing output
-The aim is to give to the user a balanced data set so that he gets about as many positive matches as false matches.- -for each suspects pairs compute different measures(e.g. jaro, initial, levenshtein...)- -if no measure is higher than a given threshold, filter the suspect pair- -one threshold per measure or one global threshold? the distribution of each measure maybe different. => One global percentile- -add an option for selecting the columns which should be match exactly, those column names should be save in the output model.- -In TDS, the user must be able to filter the data to see the pairs which are highly suspects and also pairs that are poorly suspect. (using a cursor with a min value and max value ? and also an option to choose to be inside the bounds - in [0.2, 0.85] or outside the bounds in [0, 0.2] and [0.85,1] ?-ote -Idea : give the percentile for a set of measures- -TODO : Give an API for filtering (need to compute measures and percentile)- *Note:* {color:red} The suspect output may be different with every execution of the job when set a small nb for the param "size of the sample for percentile computation".{color} A ramdon seed option (for fixing the output) will be added after TDQ-12370 be done.
Story points: 8