Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper investigates how to use auxiliary information to improve classification performance when few labeled examples are available. As the introduction makes clear, this is an especially important problem area for remote sensing applications, where labels are scarce for many inputs (e.g. satellite photos from countries/regions without much annotation).

The authors present three intuitively plausible baselines/ablations, two of which use auxiliary information, and explain the benefits and downsides of each. For instance, regarding the aux-inputs baseline: "the relationship between the aux-inputs and the target can shift significantly OOD, worsening the OOD error". These claims are later supported with theory using linear models. The theory is presented nicely, improves understanding, and is believable.

Their proposed method is very similar to the aux-out baseline/ablation. The only difference is that they fine-tune on pseudo-labeled in-distribution examples in their method. Seeing as this baseline could also be thought of as an ablation, and taking into account the improved performance of the full In-N-Out method, it is not too worrying. However, I am concerned about Remark 1 in Section 4.1, which says "We train an aux-inputs model gˆin from w,z to y on finite labeled data—since the noise σ 2 = E[ 2 ] is small this model is very accurate." If In-N-Out only improves performance when aux-in is a nearly perfect generator of pseudo-labels for in-distribution data, then doesn't this imply that aux-out would learn just as much from the GT labeled in-distribution examples? How are the pseudo-labels actually helping?

Pros:
- Underexplored, important problem area
- Good clarity of writing and paper structure, including theoretical sections
- Instructive choice of baselines/ablations
- Empirical improvements from the proposed method
- Not just vision datasets; they use a time series dataset as well
- Theory in the case of linear models to improve understanding
- The related work seems appropriate

Cons:
- The proposed method is very similar to one of the baselines/ablations, and I am not certain that there is a meaningful difference between them
- There is no comparison to previously published work that uses auxiliary information for classification. Perhaps there are no suitable baselines from prior work, but it is not clear from the paper that this is the case.

======================================================

Update after rebuttal:

The authors have addressed my concerns. Contrary to my initial understanding, the paper builds off of prior work in a methodical way, and the pseudolabeling stage of In-N-Out makes more sense now. I have raised my score from 6 to 7.
review_title: Official Blind Review #3

final_score: rating: 7: Good paper, accept
review_content: This paper introduces a new method for leveraging auxiliary information and unlabelled data to improve out-of-distribution model performance. Theoretically, in a linear model with latent variables, they demonstrate using auxiliary data as inputs helps in-distribution test-error, but can hurt out-of-distribution error, while using auxiliary data to pretrain a "good" representation always improve out-of-distribution error. The proposed method uses the auxiliary data to learn an initial model, which generates psuedolabels to fine-tune the pretrained model.

Pros:
- At a high level, this paper address a question of great interest to the ML community: out-of-distribution generalization.
- The theoretical model shows, albeit in a potentially simple linear setting, that pretraining a low-dimensional shared representation generically improves out-of-distribution accuracy.  I'm not intimately familiar with all of the papers in this area, but I think this emphasis (as opposed to transfer learning) is new. This may be of interest more broadly.
- Through experiments and a concrete example, the paper demonstrates the potential danger of using auxiliary features as input to models evaluated out of distribution.
- The paper reports experimental numbers on two real remote sensing datasets rather than solely evaluating on synthetic data.

Cons:
- Experimental results: My primary complaint with the paper is that, for the tasks considered, In-N-Out does not appear to work much better than the pretraining aux-outputs baseline. For out-of-distribution accuracy, across all of the datasets, the effect sizes are very small and the confidence intervals overlap. For in-distribution accuracy, there's only a large difference for the Landcover dataset. This makes me uncertain about the generality of the method and the potential size of the effects, though it's possible there's a more nuanced story that I'm missing. 
- Clarity: I found the description of the method confusing after several reads (section 2.1 and section 4), and the model sections are very notation heavy without necessarily providing much clarity. The graphical model, however, was very enlightening. 

Question:
- Difference between the theoretical and experimental In-N-Out models: How come the experimental procedure differs from the one that is analyzed, e.g. fine-tuning on h_out(x)? Is the performance in practice worse? More difficult to implement? I don't mind the gap, but some explanation and, if available, associated experiments explaining this would be enlightening.
- Generality of the theoretical model: How universal are the phenomenon captured by the linear model presented in Theorems 1 and 2? It's not obvious if the conclusions are "representative" or generalize beyond the ones explicitly analyzed.

==============
Update after rebuttal:

Thank you for clarifying that aux-outputs is itself a contribution and not simply a baseline for comparison. I also appreciated the additional experiment showing examples where In-N-Out can outperform aux-outputs. I'm raising my score from a 6 to a 7 accordingly.

review_title: Nice description of pitfalls and ways to leverage auxiliary data from OOD evaluation; Inconclusive experimental results

final_score: rating: 7: Good paper, accept
review_content: Theoretical and empirical study on how to combine two approaches using auxiliary information for out-of-distribution samples

**Quality:**

_Pros._ The authors propose the method to combine two representative methods (Aux-Inputs and Aux-Outputs) to exploit auxiliary information which is usually available in real-world scenarios. Beginning with theoretical analysis on Aux-Inputs and Aux-Outputs models, they show that the proposed method, In-N-Out is effective in minimizing the risk under a distributional change in a linear regression setting. Empirical validations consistently show the preference of the proposed method highlighting the improvement in out-of-distribution (OOD) samples.

**Clarity:**

_Pros._ The description of the problem setting, approach, and intuition is well-written and persuasive based on their observations in the example in Introduction. The intuitions for the theoretical findings are nicely addressed.
_Cons._ Some missing definitions (e.g., z in Alg. 1 as the output of `\hat{h}_{out}`)  and lacking rigorousness in the text make difficulty in following (e.g., the function `\hat{f}_{in}` takes two inputs, x and z, AND takes one input, `x^{id}_i`, in the 2nd and 4th lines in Alg. 1).  Figure 1 has ambiguities to understand which portion of the model is transferred and how to handle the change of the number of inputs (purely presumably, zero-filling as in the baseline in Sec 2.)

**Originality:**

_Cons._ Aside from their theoretical analysis, the combination of Aux-Inputs and Aux-Outputs is a simple model exploiting pseudo-labels (Xie et al., 2018), used for unsupervised domain adaptation tasks. In terms of novelty, the proposed method has a weak contribution. Isn't possible to borrow a sophisticated model from the works on unsupervised domain adaptation in your experiments?

**What expected in rebuttal:**

(1) Please explain the applicability of a sophisticated model from the literature in unsupervised domain adaptation in your experiments. (As an extension of the related work section.)

(2) In Sec 3., they described, "the aux-outputs model has better risk since w is lower dimensional than x. In particular, the in-domain risk only depends on the dimension but not on the conditioning of the data." However, Aux-Inputs or even a baseline (linear) model can have a lower-dimensional hidden representation in a low-rank linear model (e.g., two-layer perceptron, `f(x) := W_2 W_1 x`). So, the explanation is insufficient for the matter.

(3) The proof of Proposition 1 is incorrect. Where can we find Remark 10? There is a typo missing reference after "We use Theorem 1 in `?`." In Eqn. 23 and 24, if `1+cd/n = 2`, the left-most term in Eqn. 24 is `<= 2 \sigma^2`, but not guarantee the Eqn. 24 if `\sigma_u^2 < \sigma^2`. To satisfy Eqn. 24, `cd/n \sigma^2 < \sigma_u^2` should be true.

(4) In Lemma 8, the square is omitted inside of expectation in LHS of Eqn. 87. And, the dimension of R should be k x (k+m), not k x (k+T). I believe T is accidentally misplaced in this context.

**Minor comments:**

(5) Before Eqn. 26, a missing period right before "For the input model ..."

(6) y' and x' instead of y and x in Eqn. 32.
review_title: Theoretical and empirical study on how to combine two approaches using auxiliary information for out-of-distribution samples

Meta-review: The paper addresses the problem of improving generalization when few annotated data is available by leveraging available auxiliary information. The authors consider the respective merits of two alternatives: using auxiliary information as complementary inputs or as additional outputs in a multi-task or transfer setting.  For linear regression, they show theoretically that the former can help improve in distribution error but may hurt OOD error, while the latter may help improve OOD error. They propose a framework for combining the two alternatives and show empirically that it does so on three different datasets. 


All the reviewers agree on the novelty, interest and impact of the method. The rebuttal clarified the reviewers’ questions. I propose an accept.

Decision: Accept (Poster)