RETHINKING LOCAL LOW RANK MATRIX DETECTION:A MULTIPLE-FILTER BASED NEURAL NETWORK FRAMEWORK

Abstract: The matrix local low rank representation (MLLRR) is a critical dimension reduction technique widely used in recommendation systems, text mining and computer vision. In MLLRR, how to robustly identify the row and column indices that forma distinct low rank sub-matrix is a major challenge. In this work, we first organized the general MLLRR problem into three inter-connected sub-problems based on different low rank properties, namely, LLR-1C, LLR-1, and LLR-r.  Existing solutions on MLLRR all leverage problem-specific assumptions and mainly focused on the LLR-1C problem, which lacks the capacity to detect a substantial amount of true and interesting patterns generalizability and prohibits. In this work, we developed a novel multiple-filter based neural network framework, namely FLLRM, which is the first of its kind to solve all three MLLRR problems.We systematically benchmarked FLLRM with state-of-the-art methods on an extensive set of synthetic data, empowered by a robustness evaluation of parameters and theoretical discussions.  Experimental results showed that FLLRM outperforms all existing methods and enables a general solution to all the three sub-problems. Experiments on real-world datasets also validated the effectiveness of FLLRM on identifying local low rank matrices corresponding to novel context specific knowledge.

Reviews and author responses:

The authors propose a multi-filter based neural network framework for local low-rank matrix approximation problems. They three sub-problems, namely, LLR-1C, LLR-1, and LLR-k. Based on extensive simulations, the authors claim that the proposed method is more general and outperforms previous methods on this problem.

1. As the authors analyzed, this problem is tackling topically-coherent sub-matrix discovery in a given matrix. In Anchor-based methods, this problem is called anchor point selection. To tackle this problem, the proposed method is composed of 6 steps described in Figure 2. Some of these steps, however, are not fully justified to be present.

 1-1) The first step slices the matrix into multiple P * P sub-matrices, but it is not justified why the submatrix must be a square matrix.

 1-2) At the second step, multiple Q * Q matrices are sampled from the previous step. Again, squareness is not justified. Also, it is not justified why matrix slicing and random sampling should be in two separate steps, which are done at once in previous works. 

 1-3) In matrix slicing and sampling, it is an important criterion if the sampling allows overlapping between different submatrices. This is also not clear in this manuscript. 

 1-4) It is not clear how the matrix is sliced in the first step. In anchor-based methods, for example, a kernel is used to gather similar rows and columns. How does this work in this framework?

 1-5) After the six steps, it seems the outputs are still P * P matrices. How do we recover X from these? This is not clearly described in the paper.


2. The proposed algorithm is quite arbitrary, especially with the local low-rank filters. Previous anchor-based methods like LLORMA learn those filters (called kernels there) from the data. In this paper, however, the authors use somewhat arbitrary predefined filters like {3,2,1,-1,-2,-3,0,0}. Effectiveness of these filters should be different from dataset to dataset, so it is hard to justify generalizability.


3. Experiments can be significantly improved in many aspects.

 3-1) The authors mentioned they used two real-world data, but it is hard to find the details of those datasets. What datasets were used, how big/sparse are they, what pre-processing was done, what are the baseline performance on them? We expect reproducible details on these datasets for a successful ICLR submission.

 3-2) Regarding the scalability, it is hard to say that the scale experimented in this paper justifies "scalability". The authors used a machine with 16GB ram, which even does not fit for a large-scale dataset to be loaded. LLORMA, for example, was evaluated on MovieLens 10M with a 70K * 10K matrix, or Bookcrossing dataset (100K * 300K). These are far larger than the scale used in this paper, such as 4000 * 2000. (Please see https://www.jmlr.org/papers/volume17/14-301/14-301.pdf for the journal version of LLORMA paper.)

 3-3) In this sense, the authors' claim that "LLORMA failed to converge under this data scale" in Sec 4.1 is not trustworthy. The authors are encouraged to play with more hyperparameters on a machine with higher capacity. As LLORMA already has shown it is scalable to at least a few times larger matrices than what the authors tried, so the authors should be able to reproduce it.

 3-4) The JMLR version of LLORMA paper describes some primitive anchor point selection methods. We encourage the authors to compare this against them, if applicable. (I am not sure if they are good enough to be a strong baseline though; those methods are really primitive, something like using k-means clustering.)


4. Typos and mistakes:

 4-1) At the beginning of Sec 3.1: {E_1, ... E_M_r} \times  {F_1, ..., E_N_r} <-- the last E should be F. Same mistake in Algorithm 1.

 4-2) In the first paragraph of Sec 3.1, (ii): "R small between ..." --> "r small between". Mathematical variables must not be capitalized.

 4-3) At the end of page 5: {1000,100,10,1,1} --> 1 is repeated twice. Same for s_{MS}: 50 is repeated.

 4-4) Third line of page 6:anda --> and a - The Conclusion section is on the 9th page, which violates the page limit.

 4-5) In many places, mathematical variables are used without definition. E.g., what is E, u_k,  in Eq. (2.1)?


Overall, this paper is tackling an important problem and we encourage the authors to keep working on this, but the current manuscript is not fully ready to be published in ICLR this year.

3-3) In this sense, the authors' claim that "LLORMA failed to converge under this data scale" in Sec 4.1 is not trustworthy. The authors are encouraged to play with more hyperparameters on a machine with higher capacity. As LLORMA already has shown it is scalable to at least a few times larger matrices than what the authors tried, so the authors should be able to reproduce it.

Response: We took the reviewer's suggestion and successfully run LLORMA on the high performance computing cluster of our institute. We note the time consumption of LLORMA depends on the sparsity of the data. However, for the LLR-1 and LLR-r problem, the input data are all dense data. The performance of LLORMA is good on the sparse real-word data.

3-4) The JMLR version of LLORMA paper describes some primitive anchor point selection methods. We encourage the authors to compare this against them, if applicable. (I am not sure if they are good enough to be a strong baseline though; those methods are really primitive, something like using k-means clustering.)

Response: Intuitively, the anchor point selection methods can only detect anchor points of a certain data pattern that is identifiable from the whole data or based on additional information. However, the LLR-1 and LLR-r property is not identifiable from the whole data. Hence we consider LLORMA can only detect the LLR-1 and LLR-r patterns under the guide of other information, such as the LLR-1C property. Our experiments also suggest that LLORMA cannot detect the LLR-1 pattern when the mean of the pattern is close to the background.

4. Typos and mistakes:
We want to thank the reviewer again for the careful review of our manuscript. We have corrected all the typos and mistakes.


Response: We agree that allowing different submatrices to overlap is important to an anchor-based method. In our method, (1) our method can conduct multiple matrix slicings on the input matrix, hence the matrix slices generated from different runs can overlap, (2) the Q * Q matrices randomly sampled from each matrix slice can overlap, and (3) the overlap of the matrix slicing will not affect the performance of our method very much. Our method directly predicts the local low-rank property of each Q * Q matrix. The row and column indices of a true local low-rank matrix will be more commonly seen in the Q * Q matrix with a predicted low-rank pattern. Our method has a step to reconstruct the local low-rank matrices by using the frequency of the row and column indices observed in the same Q * Q matrix with a predicted low-rank pattern.

1-4) It is not clear how the matrix is sliced in the first step. In anchor-based methods, for example, a kernel is used to gather similar rows and columns. How does this work in this framework?

Response: Similarly, our method is fundamentally different from an anchor-based method. We provided the detailed sub-algorithms of the matrix slicing with theoretical supports.

1-5) After the six steps, it seems the outputs are still P * P matrices. How do we recover X from these? This is not clearly described in the paper.

Response: We apologize that this part was not clearly introduced in the first submission. As discussed above, our method predicts the local low-rank property of each Q * Q matrix. The row and column indices of a true local low-rank matrix will be more commonly seen in the Q * Q matrix with a predicted low-rank pattern. Our method has a step to reconstruct the local low-rank matrices by using the frequency of the row and column indices observed in the same  Q * Q matrix with a predicted low-rank pattern. The details of this part is provided in the APPENDIX of the new submission.

2. The proposed algorithm is quite arbitrary, especially with the local low-rank filters. Previous anchor-based methods like LLORMA learn those filters (called kernels there) from the data. In this paper, however, the authors use somewhat arbitrary predefined filters like {3,2,1,-1,-2,-3,0,0}. Effectiveness of these filters should be different from dataset to dataset, so it is hard to justify generalizability. Experiments can be significantly improved in many aspects.

Response: Our filters are fundamentally different from the kernels in LLORMA, instead, it is more similar to linear basis representation learned from the method. Since the local low-rank pattern is unstructured, for any filter, such as {1,1,1,-1,-1,-1,0}, its full permutation will be used to ensure a good coverage to the Q dimension. We agree with the reviewer that " Experiments can be significantly improved in many aspects.". We added two lemmas (lemma 2-3) and experiments to optimize the filter set in our revised version. We also discussed the rationale and optimization of the selected filter set in the revised version.

3-1) The authors mentioned they used two real-world data, but it is hard to find the details of those datasets. What datasets were used, how big/sparse are they, what pre-processing was done, what are the baseline performance on them? We expect reproducible details on these datasets for a successful ICLR submission.

Response: We apologize that we truncated our real-world data experiment in our first submission due to the 8-page limit. We added detailed real-world experiments, data, and analysis information in the revised version. We also provided our codes in an anonymous github link: https://github.com/ICLRanonLab.

3-2) Regarding the scalability, it is hard to say that the scale experimented in this paper justifies "scalability". The authors used a machine with 16GB ram, which even does not fit for a large-scale dataset to be loaded. LLORMA, for example, was evaluated on MovieLens 10M with a 70K * 10K matrix, or Bookcrossing dataset (100K * 300K). These are far larger than the scale used in this paper, such as 4000 * 2000. (Please see https://www.jmlr.org/papers/volume17/14-301/14-301.pdf for the journal version of LLORMA paper.)

Response: We re-conducted all the experiments on the high performance computing cluster of our institute. We added the running time on both synthetic data and real-world data as a new metric in our revised version.



We appreciate the reviewer's effort in reading and reviewing our manuscript. We particularly appreciate that the reviewer is very knowledgeable on the topic of local low-rank matrix detection. The comments from this reviewer are constructive and we have addressed all of the reviewer's concerns by rewriting our manuscript and providing substantial information in the revised version.

First, we would like to re-emphasize here we developed a new method to solve a fundamental mathematical problem, i.e. detection of local low-rank submatrices, which has not been solved by the existing method. And our mathematical derivations and experiments clearly suggested our new method can solve this hard problem. We hope to convey to the reviewer the breakthrough and the new capabilities of this work.

In our revised version, we rewrote and reorganized the manuscript by (1) clarifying all mathematical formulations and notations, (2) putting detailed mathematical descriptions into APPENDIX, (3) adding three lemmas to support the correctness of our formulation, (4) adding the details of all sub algorithms in APPENDIX, (5) adding more systematic experiments on synthetic and experiments data, and (6) including running time on synthetic and real data as an evaluation metric. We believe our revised version has solved all the reviewers' concerns.

Please see our point-to-point response below:
1. As the authors analyzed, this problem is tackling topically-coherent sub-matrix discovery in a given matrix. In Anchor-based methods, this problem is called anchor point selection. To tackle this problem, the proposed method is composed of 6 steps described in Figure 2. Some of these steps, however, are not fully justified to be present.

Response: From the author's perspective, some local low-rank matrices may unnecessarily be a topically-coherent sub-matrix, such as the local rank r matrix (r>1). The local rank-1 or rank-r property of an unstructured sub-matrix may unnecessarily be computationally identifiable by an anchor-based method. For example, most existing anchor-based methods detect regions with distinct means as the anchor, which cannot detect the low-rank pattern with a similar mean to the background noise. Noting the limitation of the anchor-based methods, we developed our approach. We feel it is improper to evaluate our methods in the scope of "anchor-based method" or "anchor point selection". In our revised version, we provide substantial explanation and details of our method in the main text and APPENDIX. Specifically, we consider the filter-based pattern detection in our method is more similar to a convolutional neural network while the matrix slicing and sampling approach is a specific type of random projection. We also added more mathematical derivations and discussions of our methods in the revised version.

1-1) The first step slices the matrix into multiple P * P sub-matrices, but it is not justified why the submatrix must be a square matrix.

Response: We agree with the reviewer that the matrix slice is unnecessarily a square matrix. However, this setting will not affect the performance of our method very much (though it could be critical to an anchor-based method). The purpose of the first step is to ensure at least one matrix slice will enrich the largest local low-rank pattern. 

1-2) At the second step, multiple Q * Q matrices are sampled from the previous step. Again, squareness is not justified. Also, it is not justified why matrix slicing and random sampling should be in two separate steps, which are done at once in previous works.

Response: Similarly, the randomly sampled small matrices are unnecessarily square and it will not affect the performance of our method very much. We provided a theoretical derivation to support that the low-rank property can be inherited in the randomly sampled matrices. However, a possible improvement of this work is to set filters on both row and column sides, for which a squared randomly sampled matrix is more rational.

1-3) In matrix slicing and sampling, it is an important criterion if the sampling allows overlapping between different submatrices. This is also not clear in this manuscript. 


We appreciate the reviewer's effort in reading and reviewing our manuscript. First, we would like to re-emphasize here we developed a new method to solve a fundamental mathematical problem, i.e. detection of local low-rank matrices, which has not been solved by the existing method. And our mathematical derivations and experiments clearly suggested our new method can solve this hard problem, which cannot be solved by existing methods.

In our recently revised version, we rewrote and reorganized the manuscript by (1) clarifying all mathematical formulations and notations, (2) putting detailed mathematical descriptions into APPENDIX, (3) adding three lemmas to support the correctness of our formulation, (4) adding the details of all sub algorithms in APPENDIX, (5) adding more systematic experiments on synthetic and experiments data, and (6) including running time on synthetic and real data as an evaluation metric. We believe our revised version has solved all the reviewers' concerns.

Please see our point to point response below 
1. The section of mathematical formulation does not do a good job of providing a clear definition of the target problem.
Response: The target problem is a mathematical problem. Truncated SVD is the most mathematically correct and commonly utilized presentation of a low-rank matrix. In our revised version, we added more details and citations to the mathematical formulation and involved the complete notation of a truncated SVD to ensure the rigorous of the mathematical formulation. 

2- The presented algorithm is composed of many steps and the paper does not provide sufficient study/analysis to guarantee their performance. The paper also lacks a theoretical study of the proposed method.
Response: We added the details and discussion of all sub algorithms in the APPENDIX of our revised version. We added three lemmas with providing mathematical derivations to support the correctness of the most critical steps of our methods. In addition, we conducted systematic synthetic and real-world data based experiments to evaluate the performance of the core sub algorithms. Robustness analysis and optimization of parameters were also provided.

3- The experiments on real data are not convincing and I suggest the author to include further experiments with real data. 
We apologize that we truncated our real-world experiment in the first submission due to the 8-page limitation. 
Response: We provided experiments on two real-world data sets in the revised version. Our real-world data based experiment clearly suggest our method can detect local low-rank matrix with context-specific meaning from highly noise real-world data. We also want to re-emphasize the method is for solving a fundamental mathematical problem that can be applied to a broad range of data. Hence we considered the most important experiment is to justify our method can accurately detect the unstructured local low-rank matrix under general error types and other data settings, and compare our performance with other methods. Hence we focused more on a comprehensive set of synthetic data than a few sets of real-world data.

4- I also suggest to provide a discussion and elaborate the connection between MLLRR and the subspace clustering problem.  
Response: We provided some discussions of MLLRR and subspace clustering problem in the introduction and related works. A straight forward understanding is that each local low-rank matrix forms a distinct subspace cluster in the data. However, considering the page limitation and the focus of this work is to provide a capability to detect local low-rank matrix, we did not extend this topic in the further part of our manuscript.





We appreciate the reviewer's effort in reading and reviewing our manuscript. First, we would like to re-emphasize here we developed a new method to solve a fundamental mathematical problem, i.e. detection of local low-rank matrices, which has been solved by the existing method. And our mathematical derivations and experiments clearly suggested our new method can solve this hard problem, which cannot be solved by existing methods.

In our recently revised version, we rewrote and reorganized the manuscript by (1) clarifying all mathematical formulations and notations, (2) putting detailed mathematical descriptions into APPENDIX, (3) adding three lemmas to support the correctness of our formulation, (4) adding the details of all sub algorithms in APPENDIX, (5) adding more systematic experiments on synthetic and experiments data, and (6) including running time on synthetic and real data as an evaluation metric. We believe our revised version has solved all the reviewers' concerns. 

In addition, we respectfully disagree with some of the reviewer's negative comments
For the reviewer's comments "The paper is confused and hard to follow. The mathematical notation is inconsistent." In our first submission, we directly adopted the formulation in the related works and gave the full citation information. We do not think it is necessary to align the formulation in several very different existing methods to a fundamentally new method. In addition, we provided a complete notation and mathematical formulation of our method in the first submission.

For the reviewer's comments "The proposed algorithm is complex and made of several different components that are not appropriately justified. No ablation study is performed." We justified the core sub algorithms in sections 3.2 -3.4 in our first submission. Ablation study is not suitable for our method since every step is critical. Instead, we conducted the performance evaluation of the core sub-algorithms in sections 3.5 and 4 in our first submission. 

For the reviewer's comments "The experimental setup is not described appropriately(e.g. you mention a feed forward neural network in section 3.4 but don't report any parameter such as number of layers or size)." The information was provided in 3.4 in the first submission. 

For the reviewer's comments "The evaluation is limited and not very supportive of the author claims, especially for real data more experiments would be necessary.", we want to re-emphasize the method is for solving a fundamental mathematical problem that can be applied to a broad range of data. The mathematical form of the to-be-detected local low-rank matrix does not differ a lot between synthetic and real-world data. The major difference between synthetic and real-world data is background error. However, considering none of the existing methods can solve this problem, in our first submission, we considered the most important experiment should justify our method can accurately detect the unstructured local low-rank matrix under general error types and other data settings, and compare our performance with other methods. Hence we focused more on a comprehensive set of synthetic data than a few sets of real world data.

Summary: The authors propose a new framing of the matrix local low rank representation technique, identifying 3 different subproblems and claiming most existing approaches sonly solve some, then proceed to propose a novel method that solves all three, evaluating on synthetic data.

Review:
* The paper is confused and hard to follow. The mathematical notation is inconsistent (e.g. what is $dist1$, $dist2$ in related works? what is $K_\Omega^h$?). There are many grammatical errors. The proposed algorithm is complex and made of several different components that are not appropriately justified. No ablation study is performed. The experimental setup is not described appropriately(e.g. you mention a feed forward neural network in section 3.4 but don't report any parameter such as number of layers or size). 
* The evaluation is limited and not very supportive of the author claims, especially for real data more experiments would be necessary. The section on scalability lacks quantitative measures of the claimed good performances. 

Summary: While the idea could potentially be promising, a more clear explanation and more attentive evaluation are fundamental to assess the methods applicability and value.


1- The section of mathematical formulation does not do a good job of providing a clear definition of the target problem. 

2- The presented algorithm is composed of many steps and the paper does not provide sufficient study/analysis to guarantee their performance. The paper also lacks a theoretical study of the proposed method. 

3- The experiments on real data are not convincing and I suggest the author to include further experiments with real data. 

4- I also suggest  to provide a discussion and elaborate the connection between MLLRR and the subspace clustering problem. 

Meta-review: We thank the authors for detailing their answers to the reviewers and uploading a new version of the paper with more details and experiments. While the experimental section has improved in the revision, the fact that the method proposed is an ad hoc sequence of 6 heuristic steps, not supported by theoretical justification, and that the paper is hard to follow and not rigorous in its statements, remain.

For example, the authors explain in their response that "[we] developed a new method to solve a fundamental mathematical problem". If this is the case, then one would expect the mathematical problem to be rigorously formulated, and the fact that the method solves it supported by theoretical justifications.

Regarding the problem statement (section 2.2), the authors write three equations, which however are ill-defined or ambiguous. For example, equation 2.1 has an expectation. Does that mean that X is a random matrix? This is nowhere stated, and there is no expectation in the following equations. Equations 2.2 and 2.3 are about a "truncated SVD" operator, which is also not rigorously defined. Literally, the authors state in Section 2.1 that for a matrix Z, tSVD¨*(Z)=USV, where S is the "diagonal matrix Z's singular values, where all except for the top r singular values are forced to be zero". But what is r? Is it a parameter of tSVD*? Or is it, as suggested in the last sentence, "the numerical rank of Z" that suggests that it depends on Z, but in that case the "numerical rank" should be defined if it is different from the rank. I take these examples to highlight that the authors should consider writing rigorous and correct equations to define the problem.

Regarding the fact that the method proposed solves the problem, the authors add in the new versions some lemmas to support the claim. Alas, these lemmas also lack rigor (and therefore correctness) in their statement and proofs. For example, in Lemma one, the statement mentions a mysterious "if the other r_{MS,i} are large enough" (what is "large enough"), mention "by expectation at least one matrix slice.." (what does the "by expectation at least" mean?), and claim as main result "at least one matrix slice [...] will enrich the matrix" (what does "enrich" mean?). Looking at the proof of Lemma 1, it is just based on a probabilistic argument that if you randomly pick enough rows or columns in a matrix, they will hit a given subset with some probability. However, the authors seem to forget that randomly selecting rows and columns is only one step in their algorithm, and that the output of "Matrix_Slicing" is obtained by subsequent steps (maximizing inner products etc...). In conclusion, the statement of Lemma 1 is not rigorous, and its proof is also vague and not correct. More generally, this lemma and the following ones are far from providing evidence that the method proposed is likely to solve the problem.

While the method may be a heuristic approach with some empirical merit, we therefore believe that the paper is currently not ready for publication.



Decision: withdrawn-rejected-submissions