Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper introduces a method for converting sentence pair classification tasks and sentence regression tasks into span into span extraction tasks, by listing all the possible classes (entailment, contradiction, neural) or the discretized scores (0.0, 0.25 ...) and concatenating them with the source text. With this formulation, one can train a BERT-based span-extraction model (SpEx-BERT) on classification, regression, and QA tasks without introducing any task-specific parameters. The purposed SpEx-BERT model achieves moderate improvement (0.3 points) over the BERT-large baseline on the GLUE test set when fine-tuned on intermediate STILTs tasks (Phang et al., 2018).

Strengths:
- Extensive finetuning/intermediate-finetuning experiments on a range of NLP tasks.
- The paper is mostly well-written and easy to follow.

Weaknesses:
- This paper presents a lot of experiments. But it seems that the most useful / head-to-head comparison against the BERT model are the last 2 rows in Table 2 with the GLEU results, where the improvement is moderate.
- The idea of expressing various NLP tasks (including textual entailment and text classification) as question-answer has been well-explored in decaNLP (McCann et al., 2018). It would be nice if the authors could elaborate more on how the proposed method differs from theirs.

Other comments/suggestions:
- Likely typo in abstract: "fixed-class, classification layers for text classification"

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This submission proposes a span-extraction approach to unify QA, text classification and regression tasks. The extensive empirical studies presented in this paper demonstrate that the proposed method can improve the performance by a small margin across multiple datasets and tasks. Overall, I find that the idea of unifying QA, text classification and regression is interesting by itself, but the experiments cannot justify their claims well mainly due to the mixed results.   

I have the following concerns:

0. Compared to decalNLP (https://github.com/salesforce/decaNLP), this new approach seems unable to handle as many types of tasks as decalNLP. It is not clear to me what is the main advantage. As discussed in the Related Work Section, decalNLP needs to fix the classifier a-priori, but this submission's method needs a natural language description, which seems more difficult to implement in practice. 
1. The results are mixed. For example, SpEx-BERT underperforms BERT on RTE and QNLI in Table 2 and Span-extractive multi-task learning results in weaker single-task models as shown in Table 4a. Furthermore, an error analysis would be helpful here. 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper asks whether it works to remove task-specific heads and treat classification and regression problems as span extraction, by formatting problems in such a way that a single span extraction model can be used.  This is a reasonable question to ask, and the authors performed a very large number of experiments attempting to answer this question.  The authors claim that using span extractive models instead of task-specific heads yields improved performance over separate heads.

My main concern with this work is actually with something that would otherwise be a strength - the very large number of experiments.  Looking at the results tables, I come to a different conclusion from the authors: there does not appear to be a significant difference between using a single head or using multiple heads (this is still an interesting conclusion).  The numbers presented all appear to be within the bounds of random fluctuations, which are not controlled for at all with standard statistical testing methodologies.  And with the very large number of experiments, there are bound to be some that stand out.  This is especially true given the methodology used for small datasets - even if it was used by prior work, it is still not statistically sound to take the max of 20 samples from a training distribution and report the difference without any other statistical testing.

As an example, look at table 3b.  This is claimed as a big win for SpExBERT when comparing the MNLI fine-tuned version.  But if you look at the other rows in the table, SpExBERT is worse than BERT by a *larger margin* than it is better in the MNLI case (contradicting the claimed result from table 2).  And this general trend is seen across the tables - the differences are small and inconsistent, making it look very much like we are just seeing random fluctuations.  This must be controlled for statistically in order to make any valid conclusions.  The one possible exception here seems to be SST.  Those results on the dev set do indeed seem to be more consistent, which is interesting, and hints at the utility of using "positive" and "negative" as natural language descriptions, as the authors claim.  It's not very convincing, however, as the test set difference is very small, and SpExBERT had more opportunities to find a good dev set number, as it had more experiments.

My second major concern is with the experimental set up.  The authors want to claim that using a unified span extraction format yields superior performance to having separate heads.  But there are baselines missing to really demonstrate this claim.  The multiple head setup isn't really evaluated as a baseline in most of the experiments (e.g., using SQuAD / other QA datasets as an intermediate task in table 2, using BERT for any of the QA datasets in table 3).  So, even if the above issue of statistical testing were solved, it would still be very hard to evaluate the claim, as the proper comparisons are not present in the majority of cases.

My main conclusion from reading this paper is that it does not appear to matter what head you use for these particular datasets.  This is an interesting result, though it is not the one that is claimed in the paper.  I think it would be very challenging to extend this approach to a broader set of tasks, however, as the authors suggest towards the end of the paper.  How do the authors propose handling cases where it is not feasible to put all possible outputs in the context?  This includes any generative task (including generative QA) and any kind of structured output (like parsing, tagging, etc.), or a regression task that does not lend itself well to bucketing.

Minor issues:

I would be careful about claiming that you've successfully captured regression in a span extraction format.  You have one regression task, and it's one where the original labels were already bucketed, so the bucketing makes sense.  I am very skeptical that this would actually work for more general regression.

Re the paragraph titled "SpEx-BERT improves on STILTs": Note that SpExBERT requires additional data preprocessing and hand-written templates when switching between tasks, which is not necessary in the method of Phang et al.  There are always tradeoffs.  Neither using separate heads nor doing your additional processing are very hard, so I wouldn't make as big a deal out of this as you do.  If you want to talk about one of the methods using more "human inductive bias", or whatever, the hand-written templates in yours might actually be _more_ "incidental supervision" than a very small difference in model architecture.  But again, the difference here is very small, not worth noting in the paper.
review_title: Official Blind Review #1

Meta-review: (I acknowledge reading authors' recent note on decaNLP.)

This paper proposes a span extraction approach (SpExBERT) to unify question answering, text classification and regression. Paper includes a significant number of experiments (including low-resource and multi-tasking experiments) on multiple benchmarks. The reviewers are concerned about lack of support on author's claims from the experimental results due to seemingly insignificant improvements and lack of analysis regarding the results. Hence, I suggest rejecting the paper.
Decision: Reject