Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents UViM, a "unified modeling approach for vision using learned guiding codes". More specifically, the main motivation of the paper is to unify the highly specialized and task-specific architectures that are present in typical computer vision tasks such as segmentation, depth estimation etc. To this end, the paper proposes a two-stage procedure inspired from NLP, where in the first stage a simple feedforward model $f: (x, y) \mapsto y$ with a discrete bottleneck in the $y$ data stream is learned. Here, $y$ denotes the target modality, e.g. a segmentation map, and $x$ is a natural image. In the second stage, the distribution of guiding codes is learned using a standard language model (encoder-decoder transformer), such that the final prediction model $f: (x, LM(x)) \mapsto y$ does only depend on an input image $x$. The model is trained on three different tasks (colorization, monocular depth estimation and panoptic segmentation) and achieves reasonable performance on all of these. 
strengths_and_weaknesses: **Strengths:**

- This is a good paper: the motivation to unify models for different visual tasks is reasonable, and the approach presented here is a good step in this direction. It is very well written and easy to follow thanks to its good structure. I agree with the authors' statement that UViM is a "brave new prototype of the general-purpose learning approach for computer vision." (l.355).
- The paper performs lots of informative ablation studies (Sec.4), although some open questions remain.

**Weaknesses:**

- A drawback of the method is that it requires paired training data, and the LM backbone has to be trained on discrete representations of the target modality (which requires pixelwise supervision). Besides the colorization task, these supervisory data is usually expensive to gather, which limits the size of the training set. Since the latent representation is modeled with an autoregressive language model, this approach is prone to overfitting (is this why dropout on the representations is required?). It would be interesting to try equally general, but potentially more data-efficient approaches (e.g., discrete diffusion models [1,2]) here.
- l.127-128, "For historical reasons we equip Ω with an additional input x, though it appears not to affect the resulting model" are unclear to me. Does this mean that the final model is always trained with this stream but it does not hurt to remove it? 
- Is the dependency on guiding code capacity (controlled via sequence length and codebook size) the same across modalities (cf Fig.6 which demonstrates this for the panoptic segmentation task only)
- Although performance is good, the approach does not yet reach that of more specialized model designs. I am not sure whether performance will reach that of more specialized models without using more specialized strategies to decode into the output space or architectural choices (which would argue against the ground tone of this work of "unifying" different areas)

**Mixed Comments:**
- For the colorization task: To increase fidelity, adding a (patch-wise) GAN loss to stage I training should help to improve FID and perceptual quality, similar to VQGAN.
- Does it help to retain spatial structure in the guiding codes? Since the model deals with image data, using 2D backbones to model the distribution of guiding codes could make the approach more applicable to smaller datasets.
- Towards a "true" unification: Would it be possible to train the LM on all domains simultaneously, potentially making use of cross-domain interdependencies, and use a "task-token" to signal which task the LM has to sample from?
- Does, similar to classical vision models,  a "convolutional application" work out-of-the-box? E.g., can we apply the model on larger resolution than it was originally trained on? I think this depends highly on the implementation of the positional encodings of the ViT backbone, would be good to add some explanation here.
- How important is the scale of the pre-trained encoder (the only comparison is to "from-scratch") ? Can it be fixed?
- Side question: Am I right that the model can be interpreted as a variant of "ViT-VQGAN" (with task-specific reconstruction loss) which is trained on "labels" instead of images but conditioned on the latter? 

In summary, this is a solid paper, with  room for exploration and improvements in the future. I am recommending it for publication at the conference.


_References_
- [1]: Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions, Hoogeboom et al
- [2]: Structured Denoising Diffusion Models in Discrete State-Spaces, Austin et al
questions: Please see the above section on "strengths and weaknesses", mainly the section with mixed comments.
limitations: Yes, limitations are briefly discussed in the conclusion ("rough edges"), and no negative societal impact is anticipated.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes UViM, a unified modeling approach for vision tasks with learned guiding codes. The approach has two stages. It first learns a discrete code from the GT to guide the prediction, and try to predict the code from the input in the second stage. Experiments on three dense vision tasks are conducted to prove the effectiveness and generalization of the method.
strengths_and_weaknesses: Strengths:

The idea is quite novel. Previously, the discrete code is mainly used for generative tasks, but in this work, it serves as a bridge between input and GT. The code is informative enough to guide the inference, since it has access to the GT in the first stage. The discrete bottleneck makes sure that the code is compact and abstract, so that it can be learned by a language model in the second stage. The autoregressive language model is flexible, making it a good choice to model the different structure of each task. 

The paper is also neatly written. It is easy to follow.

Weakness:

Even though the idea is cool, and it may work well, I don't think the experiment is enough to support the effectiveness of the method. A unified structure for vision tasks has been studied for a while. Methods like Perceiver-IO and the following works have been trying to unify vision tasks by exploiting the flexibility of transformer structure. I think a comparison with Perceiver-IO or similar works should be presented. 

The ablation study on restricted oracle model can compensate for the missing comparison, since the base model (f: X->Y) is already a vanilla baseline of unified structure. But the ablation is not fair. The LM (ViT-L/16 + BERT-Base) is much larger than the base model (ViT-B/16). It's hard to say whether the proposed method is better or it simply has more trainable parameters.

I feel sorry for having to give the paper a lower rating than the method should deserve, but the experiments can not convince me that the method is better than a vanilla baseline. If the missing experiments or a fair comparison with the base model is provided, I feel happy to change my rating. 
questions: I have no more questions. 
limitations: The limitations have been well addressed in the paper.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an approach for computer vision tasks involving large output spaces overlaying the image -- like colorization, object detection, etc. A challenge with these tasks is that, because they have large output spaces, it is nontrivial to model them in an autoregressive setting, even though such a setting works great for NLP tasks. This paper proposes to learn a "guiding code" that represents the image overlay over a latent space of discrete tokens. The authors use 256 tokens in total, so the discrete tokens are highly compressed versus the [H x W x C] overlay. 

The paper proposes a two-stage approach for training. In the first stage, the model learns the code through a VQ-VAE like setting. A 'restricted oracle' is used that gets access to the labels and must summarize it through the discrete code; the 'base model' must reconstruct the labels given the code and the input image. During the second stage, the model learns to generate the guiding codes given images (mimicing the oracle given just the image).

The paper shows strong results across a variety of different tasks, including Panoptic Segmentation (COCO), NYU Depth, and ImageNet colorization. The paper has helpful ablations that discusses what helps and doesn't in the proposed framework, including what the optimal sequence length is.

strengths_and_weaknesses: 
Strengths:
* This paper presents a very simple approach that brings together several different 'image overlay' tasks from computer vision. To the best of this reviewer's knowldege, this model is novel and creative. It will likely be of interest to many in the community given the recent successes of image generation approaches like DALL-E.
* The results are strong, especially for not being very optimized for these diverse vision tasks.
* The ablations are helpful for understanding what the model is doing, showing that (among other things) the autoregressive model helps a lot, and there is a 'sweet spot' in terms of code length.

Weaknesses: An exceptionally weak limitations / social impact section (see later), but nothing else that stands out to this reviewer
questions: 
* Is there some way to visualize the codes to get a sense of what the model is doing?
* Could it be possible for the approach (given that it is to some degree class-agnostic) to generalize even beyond the supervised data sources that it's finetuned on?
limitations: The social impact section feels very weak to this reviewer. The following is not a strong argument: "the proposed modeling approach is very general ... As a result, we do not anticipate any negative societal impact of our technique."

There is a limitations section, but it might be great to speculate a bit more about what current bells and whistles are truly needed in other NMS/Hungarian Algorithm detection/segmentation approaches, and where this approach might fit in.

I'd be willing to adjust my score if these points are better handled.
review_title: null

Meta-review: metareview: Three reviewers provided positive reviews which were further strengthened post discussion. They agreed that that the motivation was strong, the model was novel and the paper was well written. They appreciated the ablations provided by the authors and found the results compelling. The main concern by the reviewers was a missing experiment which was provided by the authors in their rebuttal, and was appreciated by multiple reviewers. In summary, the reviewers are unanimous in their support of this paper. I agree with their reviews and I recommend acceptance.
confidence: Certain
Decision: Accept