Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A new matrix factorization method is presented in this paper in order to improve the interpretability of questionnaires through bounds and sparsity constraints. The proposed method utilizes an optimization procedure with theoretical convergence guarantees to detect latent dimensionality in synthetic data. The empirical studies only applied to two datasets, including, a commonly used general-purpose questionnaire, and the Healthy Brain Network study. Overall, I found this paper difficult to follow, and I am not sure from the standpoint of machine learning how the proposed method is a novel one. Additionally, the empirical studies did not provide sufficient evidence to support the proposition that the proposed method is superior to existing approaches. Major concerns and minor comments are presented in the review section.
strength_and_weaknesses: Please refer to the review section.

clarity,_quality,_novelty_and_reproducibility: Please refer to the review section.

summary_of_the_review: 1) The performance of the proposed method is only benchmarked on two datasets, i.e., the general-purpose questionnaire, and the Healthy Brain Network study. There are several reasons why empirical studies did not provide sufficient evidence for evaluating a novel machine learning approach - such as batch effects, data collection error, missed label handling, etc. Several more datasets should be benchmarked in order to be able to trust the current results.

2) According to the data section, the Healthy Brain Network has 11 class labels. Therefore, reporting ROC-AUC (which is mainly used for binary classification) in Table 1 and Figure 3 may not be an appropriate method of evaluating the proposed method. What was the reason behind not using classification accuracy in the process?

3) The proposed method should be compared with the state-of-the-art factor analysis techniques and method techniques that are widely used for questionnaire/tabular data.

4) According to the author(s), the proposed method is more interpretable than existing approaches. The proposed method could be compared with other techniques in terms of explainable artificial intelligence (XAI).

5) In Figures 1 and 2, why validation errors are reported rather than test errors?

6) In Figure 1 Bottom Right, what is X axis metric? What are the upper and lower bounds for this axis? 

7) The current format of Figure 1 Top is not informative. Furthermore, this figure illustrates that the majority of the factors in Q are close to zero - we assume that these characteristics cannot significantly affect the prediction procedure. If we apply thresholding or regularization to push them to zero, what will happen?
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a factorization method for clinical questionnaires by by idea of matrix completion. Given an original data matrix which stacks answers of a questionnaire from all respondents. The method searches by optimizing a regularized loss function under interpretability constraints for a pair of factor and loading matrices so that the product of the factor and loading matrix recover the original matrix. In the situation that there are multiple questionnaires, a meta-factorization is proposed in this paper by first factorizing each questionnaire, and then perform a second factorization on the concatenation of all first-step factor matrices. Numerical experiment has been performed on real questionnaires.
strength_and_weaknesses: - This is a solid paper. Merits of the proposed methodology have been well substantiated by comparing with alternative methods on data as well as simulation. The choice of cross validation seems appropriate. Theoretical guarantees for the convergence of algorithm have been provided.
- Numerical experiments show significant advantage over existing methods.

Questions: 
- For meta-factorization, is it necessary to have equal number of participants $n$ for all questionnaires? Otherwise, it seems one can’t concatenate all factor matrices like in Eq. (6). If so, I wonder how one can relax this constraint? Because in practice, the number of participants may not be the same for each questionnaire due to non-compliance, so requiring the number of participants to be the same could be unrealistic.
- As a follow-up to the last question about meta-factorization, I wonder if the sets of respondents have to be the same for all questionnaires? In practice, they may be different due to non-compliance or missing data. Some participants who answered questionnaire A may not answer questionnaire B. 
- About the optimization problem (ICQF), Contribution 2 mentioned: “if this number of factors is close to that underlying the data, the solution will be close to a global minimum”. Could “the number of factor of the underlying data” be more explicitly defined? In practice, I wonder which optimization problem is more non-convex, harder to reach a global minimum and more numerical unstable: the one with number of factors over-estimated or the one with number of factors under-estimated.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written.
summary_of_the_review: Overall I think it is a high-quality paper. It is well written and its methodology has been carefully tested with numerical experiments and theoretical study. Of course, its technical quality builds on the abundance of literature on matrix factorization and related optimization techniques like ADMM and cross validation, so the methodological novelty is perhaps not its strongest part. However, I feel the method has been well demonstrated with interesting datasets and application, which deserves recognition
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present a novel factorization method for questionnaires that emphasizes preserving interpretability (ICQF). Authors provide theoretical convergence guarantees for their algorithm. The approach is validated using synthetic data with known latent dimensionality. Furthermore, authors apply their approach to real data and show ICQF can uncover a reduced set of common factors spanning multiple questionnaires. This work shows that ICQF algorithm can preserve relevant information for diagnosis and performs well compared with similar approaches.
strength_and_weaknesses: Strengths 

A sound method with some theoretical guarantees that performs well in practice and has a wide range of impactful applications.

Weaknesses
clarity,_quality,_novelty_and_reproducibility: Clarity

The paper is very clearly written. The formulation of the problem and the optimization procedure are clearly outlined. The paper clearly highlights its novel contributions and makes references to key relevant related work. 

Quality

The quality of the paper is sound. The approach is convincingly validated by using synthetic data and by comparing with other similar algorithms. The authors apply their method to real data to estimate latent factors across questionnaires and predict diagnosis, producing compelling results and providing a sufficiently representative view of the real-world applicability of the algorithm. The theoretical section of the paper aimed at demonstrating that the algorithm can converge to a global minima seems solid and there are no errors I could see, although I did not check all the math in detail.    

Novelty

The paper introduces a novel factorization method that preserves interpretability. The resulting factorization method is new but the procedures and techniques used to derive it (ADMM, sparsity inducing regularizers and so on) are not. The novelty is somewhat limited but significant nevertheless.   

Reproducibility

The authors provide enough information to enable others to reproduce their work. I have not personally attempted to reproduce the key results of the paper.
summary_of_the_review: Overall a solid contribution to the field of factor analysis and a very interesting tool for analysis of questionnaire data. In my opinion this work deserves a spot in the conference.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: On the context of interpreting questionnaires in psychology, this submission contributes a method for non-negative matrix factorization with additional constraints to further enhance the interpretability of factors.

The approach uses penalties on both matrices of the factorization and solves the minimization problem by alternated minimization with inner problems solved by ADMM.

The method is demonstrated on a psychology cohort dataset, looking at diagnostic performance with prediction from latent factors. 
strength_and_weaknesses: The latent factors seem indeed to reveal some interesting structure in the questionnaires.

However, the method is very classic and many variants of such matrix factorization have been published. 

Also, in terms of solver, the alternated minimization are very slow. Online methods are much faster, such as the original work of Mairal and later work adapting it to me complex settings (including brain imaging).
clarity,_quality,_novelty_and_reproducibility: The manuscript is reasonably clear and I see no major error.

However, I do not see what make it stand out from a vast literature of prior work developing matrix factorization or dictionary learning with various applications, including brain imaging.

In terms of reproducibility, I do not believe that code has been shared. 
summary_of_the_review: An interesting idea but that does not stand out as very different from prior efforts.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an algorithm to identify "meta-factor" from subjects' answers to multiple questionnaires with different lengths. Specific constraints are introduced and the problem is formulated a constrained optimization problem. An ADMM algorithm is derived to compute the factorization solution. Performance comparison with some existing methods have been presented and the way of using the proposed method to interpret the meta-factors extracted from the questionnaires' results was illustrated.
strength_and_weaknesses: Strengths:
+ The application scenario is clearly presented.
+ The problem is clearly formulated in general.

Weaknesses:
- The key contribution of the paper is not clear. As mentioned in the paper, there are different alternatives to organize the multiple questionnaire data matrices for the factorization. It seems that performance gained in terms of ROC-AUC together with the interpretability is due to the application of constraints to factor values and reconstructed matrices. Is it not clear what happen if similar tricks are applied to the setting with the questionnaire data matrices concatenated. Also, the way to handle missing values and the use of ADMM for constrained optimisation is not new. Overall speaking, the novelty in term of algorithm is considered incremental.
- It seems that the interpretability could be related to the control of the sparsity as discussed in Section 3.1. I noticed only that the values of p and q for controlling the sparsity are set to be 1. There seems no more investigation on sparsity afterwards.

Question:
In Section 3.1, regarding continuous confound, if one wants to consider different ranges of value, will the mirroring procedure be sufficient?

Some typos
* The variables p and q are used twice to denote different concepts, sparsity control and row and column shuffling in BCV (Section 3.1). 
* The mask defined in ICQF is not the same as the mask defined in 3.3, but the same notation is used.
clarity,_quality,_novelty_and_reproducibility: Clarity is satisfactory and the proposed algorithm is clearly presented.

As explained in "Strengths and Weaknesses" section, the novelty of the proposed algorithm for matrix factorization (a well developed field) is considered incremental.
summary_of_the_review: This paper proposed a matrix factorization algorithm which can handle multiple questionnaire data matrices together with the objective to extract some meta-factor to support psychiatry research. While the paper is clearly motivated and presented from the perspective of application, the novelty from the perspective of machine learning algorithms is considered incremental.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors present a procedure for interpretable nonnegative matrix multiplication. Two of the reviewers were positive and, three reviewers were negative. After I read the paper, I believe the positive reviewers provided superficial reviews without properly assessing it.

The loss proposed in Section 3 (bottom of page 3) is a standard square loss with standard nonnegative constraints for W and Q. The regularizers for W and Q are standard too. The only aspects that might be novel are the constraints over Z, not to provide values that are not compatible with M, and the mask \mathcal{M}. But these are minor and straightforward contributions at best. The authors mention that their procedure is interpretable. But they do not say why or what makes their algorithm interpretable.

The results in Section 5.1 for the CHILD BEHAVIOR CHECKLIST look like the ones that we could get with Nonnegative matrix factorization, which are not shown for this problem. Some factors are very clear others not so much. The results in 5.3 in which the proposed results are compared with NMF and other methods show almost no difference in AUC terms, which makes sense. 

For this paper to be accepted, there should be an apple-to-apple comparison to a standard NMF for the proposed data in Section 5.1. For example, the authors should replicate Figure 1 with other methods. From this comparison, the authors should explain why their algorithm is better. Also, if the solution is more interpretable than NMF they should indicate what makes it more interpretable. I would also compare with a model that considers the answers are discrete, e.g, http://proceedings.mlr.press/v33/gopalan14.pdf.
justification_for_why_not_higher_score: The paper is a standard Nonnegative matrix factorization algorithm.
Decision: Reject