Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper aims at solving a task called predictive querying, which aims at producing the probability that probabilistic queries such as “when will event A occur next” or “what is the probability of A occurring before B,” happens in an autoregressive (RNN/Transformer) model. The paper categorizes the probabilistic queries into five categories, and in principle, most of them are intractable with a nearly O(V^K) complexity, where V is the vocabulary size and K is the step size. The paper proposes two approaches to estimate the predictive querying, one is importance sampling, and the other is approximation within beam search. On empirical evaluation, a hybrid of these two techniques works the best on three user behavior and two language sequence datasets.

strengths_and_weaknesses: Strengths

1. The paper is the first work that studies predictive querying on modern autoregressive models, and provides effective estimation techniques including importance sampling and approximation in beam search to estimate the probability of these queries, with proven convergence.

2. On several datasets across different domains, the paper shows that the proposed hybrid method achieves the best relative errors compared to the baseline methods.

3. The paper shows that in general, the hybrid method is the best, but in lower entropy regimes such as Mobile Apps, the beam search approximation can outperform the importance sampling and hybrid method. Such a result is informative and may inspire further research.

Weaknesses

1. It is a bit unclear to me what queries and data the paper actually uses in the experiments. It would be better to demonstrate with a few text examples for each dataset.

questions: 1. Given the five probabilistic query categories, the paper only investigates the hitting time queries. Is the proposed method able to generalize to the other categories?

limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new problem for neural autoregressive sequence models: How do we query the model about probabilistic questions like "what is the probability of A occurring before B". Given a pretrained sequence model, it is usually intractable to query the model to answer those questions, so the paper proposes 2 estimation methods: importance sampling and beam search. The paper also tries to combine those 2 methods and finds that the combination of them achieves the best results.
strengths_and_weaknesses: The problem raised by this paper is interesting and was thought intractable before. The problem of token counting or the ordering of some tokens can be potentially useful for many real applications, e.g. finance and NLP.

Although the methods in this paper are existing ones, certain tricks are applied to each of them, for example, coverage-based beam-search and tail-splitting search. The hybrid method combines the advantage of sampling and beam search, which yields the best results.

The paper is well-motivated and well-written and can be valuable for many other fields. However, it does come with certain drawbacks:

1. The 2 methods used in the paper are existing ones and lack novelty. Applying them to the estimation of sequences is natural but less non-trivial.
2. The definition of "probabilistic queries" is big and only parts of them are discussed in the paper. When the Q space gets larger, the estimation gets harder under the framework of this work. For example, the Q5 cannot be easily handled when C(K, n) gets large. In fact, only hitting time (Q3) is tested in section 5.
3. Furthermore, the experiments are not sufficient and the actual metric relies on the sampling (L266). No baselines are available, which makes it hard to judge the contribution of this paper. A suggestion would be training a supervised model for each K to predict the hitting time.
4. I have a few methodological concerns that are listed in the question section.
questions: 1. Given the sequence space can be large and the importance sampling has non-negligible variances, it is helpful to show the effective sampling size.
2. How do you think of ISR (importance sampling resampling)? Resampling can significantly improve the coverage of the importance sampling and is unbiased, which can be a surrogate of the beam search.
3. How do you balance importance sampling and beam search in the hybrid method? I wonder about the ratio of their sequences and possibly their contributions to the resulting probability (i.e. L202).
limitations: The societal impacts of this work are already discussed by the authors and I don't have further concerns.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces the problem of predictive querying in neural autoregressive models, where we answer probabilistic queries such as hitting time of an event wrt the model (not some ground truth). They propose an importance-sampling based framework and proposal distribution for estimating these queries, and also combine it with beam search for better results. The provide some understanding of the failures of beam search for predictive querying, including the correlation of beam search's error with the conditional entropy of the model.    
strengths_and_weaknesses: - The paper raises an important question about how to answer probabilistic queries about future steps, which could be applied to decision making systems that require some forecasting. They also provide a framework, baselines, and an improved hybrid method, as well as some understanding of why beam search approaches are worse. The paper is well written overall, though the explanation/motivation of the proposal distribution could be made clearer.
- On originality, while the setup seems new for neural autoregressive models, the family of problems is well studied in the stochastic processes literature, survival analysis, etc, but as far as I know they usually assume some parametric form for the underlying probabilistic model / hazard function / etc. that is easier to work with. For neural models there is less structure to work with. 
- The paper shows a nice opposite behavior for beam search and Importance sampling methods, and proposes a hybrid approach that nicely combines the two for mutual benefits, using importance sampling to estimate the gap between the beam search lower bound and the true probability.
- The paper considers a variety of sequences (text, interactions, usage records) and models (LSTMs, GPT2 (which one?) ).
questions: - The paper discusses coverage-based and tail splitting beam search (and perhaps claims it as a contribution?) but doesn't show its results, saying that it did not do well. Perhaps this is an extraneous part of the paper?  
- While hitting time is a component of other queries, it could be interesting to see how the error compounds with multiple related queries. 
- One limitation is that the evaluation method requires an expensive computation of the true probabilities. Is there a different way to verify the performance of the query estimation on longer horizons? While it's clear that the IS approach is better than beam search, it's unclear how far it can go. One possibility is to sample sequences from the models where an event actually occurs, truncating this sequence K steps before the event occurs, and then evaluating the likelihood that the event occurs under the predictive query from the past (and average over many sequences).
limitations: They discuss the potential biases of using forecasting models for decision making, especially on data like student interactions as used in the paper. They also preprocess the datasets to not contain personally identifiable information.
review_title: null

Meta-review: metareview: The paper tackles an important question: how to answer probabilistic queries about future steps, such as “when will event A occur next”, in the context of autoregressive neural models that are widely used in many applications nowadays. While similar questions have been studied in stochastic processes, it has not received much attention in inference on autoregressive models. This paper formulates the problem, provides a framework, develops baselines, and proposes improved method based on importance sampling and beam search. All reviewers agree that the paper opens an interesting problem space and is technically solid. Therefore, I recommend acceptance.
confidence: Certain
Decision: Accept