Project: talenddataquality
Issue TDQ-9320: Fix the out of memory issue when running match analysis
Implement RecordIterator for database resultset and file delimited, use iterator instead of java collection when running the analysis in both VSR and swoosh algorithm. SKype discussion: {quote} [5:47:09 PM] Sebastiao Correia: oh, I see that VSR has also the same issue [5:47:29 PM] Sebastiao Correia: maybe, we don't need François [5:48:03 PM] 刘心泉: yes, when I test on my computer, vsr have the same issue with t-swoosh [5:49:54 PM] Mingli Zhao: There are two kind of tests till now (with Liu's help) 1. Set memory limit in studio launch configuration (-Xmx 1024), there will be out of memory issue, after analyzed with MAT, Liu summarized the place where the problem is on JIRA. this issue happens both on VSR and T-Swoosh 2. NOT setting the memory limit in studio launch configuration, it will be "GC overhead limit exceeded" like I wrote on JIRA. In this case the MAT cannot profile the file which is about 6 GM size . [5:50:28 PM] Sebastiao Correia: ok [5:50:43 PM] Sebastiao Correia: the error does not seem related to the algorithms here [5:50:56 PM] 刘心泉: yes, there a point need to be attention, my computer is windows it maybe different with linux(mzhao's computer) [5:51:14 PM] Sebastiao Correia: it rather seems to appear when we read the result set of a database connection [5:51:52 PM] 刘心泉: for test 1: handleRow(oneRow); [5:52:06 PM] 刘心泉: will have problem: [5:52:09 PM] Sebastiao Correia: ah yes [5:52:16 PM] Sebastiao Correia: it will store all rows in memory [5:52:24 PM] Sebastiao Correia: in dataFromTable [5:52:30 PM] 刘心泉: protected void handleRow(Object[] oneRow) throws Exception { if (storeOnDisk) { storeOnDiskHandler.handleRow(oneRow); } else { dataFromTable.add(oneRow); } } [5:52:31 PM] 刘心泉: yes [5:52:38 PM] 刘心泉: store in a List [5:54:30 PM] Mingli Zhao: yes, that's the current way we compute it. [5:55:05 PM] Mingli Zhao: maybe we can improve it by handle row by row without store it when run the analysis. [5:55:37 PM] Mingli Zhao: but we have to consider the case when "block key" is set. [5:56:11 PM] Sebastiao Correia: yes, I think it can be improved [5:56:48 PM] Sebastiao Correia: the component had been improved a lot some time ago in order to avoid storing everything in memory [5:56:58 PM] Sebastiao Correia: I think we should at least do the same [5:57:17 PM] Sebastiao Correia: If I remember well, Yin had worked on the component optimization [5:58:44 PM] 尹悦燕(YinYueyan.talendbj): :P the last component optimization was different with this case [6:01:08 PM] 尹悦燕(YinYueyan.talendbj): agree with zhao's suggestion [6:01:30 PM] Mingli Zhao: (note that if we want to handle it row by row, we need to refactoring the way blocking key computation done, from BlockingKeyHandler.run()) [6:02:05 PM] 尹悦燕(YinYueyan.talendbj): oh, if one block is very big... [6:02:17 PM] Mingli Zhao: we can see now the data is set once by "setInputData(List inputData)" in BlockingKeyHandler [6:02:36 PM] Mingli Zhao: we can handle row by row here maybe [6:02:50 PM] 尹悦燕(YinYueyan.talendbj): ;) maybe, we can try [6:03:20 PM] Mingli Zhao: then we don't need the list property dataFromTable then in SQLExecutor. [6:04:16 PM] Sebastiao Correia: François used an iterator in the t-swoosh algorithm [6:04:39 PM] Sebastiao Correia: I don't know if we can also use an iterator here... [6:04:53 PM] Sebastiao Correia: but we should avoid to have all data in a list for sure [6:05:29 PM] Mingli Zhao: Yes, in MFB: execute(Iterator sourceRecords) [6:08:48 PM] Sebastiao Correia: maybe that is too big a refactor now [6:09:30 PM] Mingli Zhao: yes, i think so. [6:09:36 PM] Sebastiao Correia: I advise to work on a branch for this issue and only push on 5.6 if we are really sure not to introduce regressions [6:11:34 PM] Sebastiao Correia: I suggest to try and find a possible solution first. Then we can decide whether we have the time to implement it. [6:12:37 PM] Mingli Zhao: ok, this is for the memory issue of 1st test. [6:12:43 PM] Mingli Zhao: about 2nd: [6:12:50 PM] Mingli Zhao: we need to find the root cause still [6:13:24 PM] Sebastiao Correia: I think the cause is the same [6:13:45 PM] Mingli Zhao: actually the "Iterator" itself requires the data store in memory first, no? [6:14:44 PM] Sebastiao Correia: "GC overhead limit exceeded" may happen when the garbage collector is trying to free memory too often. [6:15:15 PM] Sebastiao Correia: in our case, because the memory is full or almost full because of the list, the GC will try to free other objects [6:15:31 PM] Sebastiao Correia: at one point, it cannot free the memory, hence this exception [6:16:06 PM] Sebastiao Correia: in the first case, we run out of memory immediately because the GC has not even the time to try and free the memory [6:16:19 PM] Sebastiao Correia: [12:13:46 PM] Mingli Zhao: actually the "Iterator" itself requires the data store in memory first, no? [6:16:24 PM] Sebastiao Correia: not necessarily [6:16:46 PM] Sebastiao Correia: when you iterate over a sql.ResultSet, not all rows are loaded. [6:16:57 PM] Sebastiao Correia: they are loaded in a lazily manner [6:17:46 PM] Sebastiao Correia: here, if I understand, we have all the rows in memory. Am I right? Or do we have only one block of rows in memory? [6:18:04 PM] Sebastiao Correia: (by block, I mean a block created by the blocking key) [6:18:17 PM] Mingli Zhao: all rows in memory [6:18:52 PM] Sebastiao Correia: ok, this needs to be changed somehow [6:20:05 PM] Sebastiao Correia: when we use the store on disk option, do we have this memory issue? [6:20:50 PM] Mingli Zhao: i didn't test this case but i think it will be. [6:21:11 PM] Mingli Zhao: because the MFB algorithm accept a iterator [6:21:30 PM] Mingli Zhao: currently I use the List as the collection where iterator iterates . [6:21:50 PM] Mingli Zhao: how can we construct a java.util.Iterator in order to pass this param to MFB algorithm (not use java collection)? [6:23:10 PM] Mingli Zhao: http://docs.oracle.com/javase/1.5.0/docs/api/java/util/Iterator.html [6:23:30 PM] Mingli Zhao: it works only in the java collections framework probably [6:24:40 PM] Sebastiao Correia: RecordIterator ? [6:25:20 PM] Sebastiao Correia: François implemented his own iterator [6:25:30 PM] Sebastiao Correia: I don't know whether we could reuse it [6:26:02 PM] Sebastiao Correia: otherwise, we are now expert in MapDB ;-) that could be a way to solve it [6:26:25 PM] Sebastiao Correia: but, I recommend you to first look at what did François [6:27:38 PM] Mingli Zhao: in RecordIterator , there is a list as well: List rcdGenerators [6:29:20 PM] Mingli Zhao: ok, maybe I got a idea now to wapper the jdbc resultset into RecordIterator .... [6:29:42 PM] Mingli Zhao: when iterator iterates a row, we call resultSet.next()... [6:30:12 PM] Sebastiao Correia: yes, that could be a solution [6:30:33 PM] Sebastiao Correia: finally, don't hesitate to contact François about this. [6:30:33 PM] Mingli Zhao: this requires the interator strictly "bound" to the data source we profiling [6:31:05 PM] Sebastiao Correia: yes, I think this should be ok with MFB. [6:31:17 PM] Sebastiao Correia: I'm wondering whether we can simply do the same for VSR... [6:32:16 PM] Mingli Zhao: it could be [6:33:16 PM] Mingli Zhao: ok, let me summarize the discussion tomorrow on JIRA for this performance issue. [6:33:26 PM] Mingli Zhao: but we may have no enough time to accomplish it in M4. [6:33:49 PM] Sebastiao Correia: ok [6:34:16 PM] Sebastiao Correia: don't start big changes now. Let's see if we can first find a good solution {quote}
Story points: 5