Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:
The paper proposes a PMF variant that replaces the free-form representation for latent factors from explicit mapping from inputs, to improve interpretability. A reasonable probabilistic formulation with carefully selected priors is provided, but inference is carried out using generic tools, and the method is illustrated on artificial data and a simple comorbidity application.

Reasons for score:
The proposed model is interesting and well motivated, and the technical details regarding the choice of priors for encouraging sparsity are good and match current recommendations. The model itself is a bit counter-intuitive, explaining a generally desirable property (latent variables following a reasonably chosen prior distribution, free from additional computational constraints) as a limitation and proceeds to replace it with a simplified mapping from inputs, but as the mapping itself is well justified in terms of sparsity the overall construct still makes sense. The presentation angle is somewhat narrow and some connections are missed (e.g. the authors do not explain this as amortizing the inference for the latent variables, but explain the model in terms of encoders/decoders), but this is not a major issue.

My biggest issue with the paper concerns inference. The description is limited, only referring to a specific algorithm (ADVI) without specifying all details (mean-field vs full-rank approximation), and there is no discussion or analysis on how well it works. The authors do say that the algorithm converged fast, but do not show this in any experiment. More importantly, the convergence does not yet guarantee the approximation is good and there are well-known cases for which ADVI does not really work that well. Expanding both the discussion and empirical demonstration of this would be critical. Now you say "one may use..." and "one can access..." with references to specific techniques for evaluating the quality, which gives the impression you have not actually done that.  As I presume you implemented the model in Stan (no point in using ADVI if not; there are better stochastic VI methods around that are also easier to implement), would you be able to compare the inference results against HMC at least in some small-scale problem?

Pros:
1. The idea is insightful and matches well the application needs.
2. The priors match current literature on suggestions for sparsity-inducing priors.

Cons:
1. Very limited coverage of inference, which is an important aspect even if carried out by an external software. Both theoretical and empirical evidence is missing, even though methods for evaluating the approximation quality are referred to.
2. Figures 3 and 4 are pretty, but the spherical representation does not seem to add anything here and only results in the plots taking too much space while being slightly more difficult to read. More generally, the comorbidity example is a bit superficial and could have been developed a bit further. 

Questions:
1. How was ADVI applied?
2. How did you check the approximation is good? Did you apply WAIC/PSIS-LOO or just say that it could be done?

Modifications after discussion:
Increased score by one since the revised paper clarifies the missing details on inference and also improves the motivation.
review_title: Interesting HPF variant with slightly superficial presentation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This manuscript revisits the hierarchical Poisson matrix factorization (HPF) promoting sparsity in both the representation and the decoding function with a Horseshoe+ prior. Sparsity on both sides is put forward for interpretation purposes, and to create a column-clustering property. In addition, the proposed approach caters for non-linear decoding using a GAM model.

The main practical benefit compared to sparsifying priors (as in Gopalan et al. (2014)) is that the contributed method can be solved with automatic differentiation variational inference and hence stochastic solvers, which makes it in theory easier to scale, though this improvement is not demonstrated empirically in the manuscript.

The manuscript is well written: precise and well articulated. It develops well the theoretical point that sparsity in encoding and decoding is important. However, the practical value of the contribution is not strongly demonstrated. In the real-life application, on comorbidity data, the sparsity is demonstrated as expected, but the benefit compared to other approaches is difficult to gauge, whether it is to a non-sparse approach, or an approach based on sparsifying priors. The benefit of the GAM decoding is not clear.

While there are good theoretical arguments for the model, they only partly convince in practical terms: a full analytics pipeline has made aspects to it, and the arguments might not be as important as they seem in practice. It could help to perform more empirical comparison, and to study more the contribution in the context of full analyses. The empirical demonstrations show that the model exhibit the properties that it was designed for: sparsity in the decoding. What are practical consequences of these properties in real-life applications?

In the bigger picture, it is unclear to see how this contribution positions itself in terms of practical benefits in the vast literature on latent factors with distangling approaches (distangling autoencoders, various matrix factorizations including NMF with different losses).


review_title: Good theory, not completely convincing in practice

final_score: rating: 6: Marginally above acceptance threshold
review_content: ** Description

This paper provides a new approach for finding a sparse encoding of count data matrices and hence automatically achieve feature selection.

** Pros

The proposed technique is clearly efficient and practical.  It identifies a failing in traditional hierarchical Poisson matrix
factorisation (HPF) and proposes a solution.  This approach is tested on real world datasets where its usefulness is demonstrated.

** Cons

For me the weakness of the approach is that the solution proposed appears very ad hoc with no probabilistic basis.  Particularly in
matrix factorisation problems where there is often no easy way to establish ground truth: this often leads to setting hyperparameters arbitrarily.  To be fair the authors point to ways of assessing predictive power which might help in this domain.  A less ad hoc model might allow a more principled approach to choosing hyperparameters.

review_title: Sparse Encodings for counting matrix factorisation.

Meta-review: The authors present a hierarchical factorization of the Poisson matrix and explain why sparcity in the encoder is important for interpretability. The reviewers appreciated the contribution of the paper and highlighted the advantage of such an approach for users. The authors have improved their initial version by adding more detail on inferences and experiments.  The decision is to accept the paper.
Decision: Accept (Poster)