Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a domain-independent data augmentation operator for regression tasks: $\texttt{comfort-zone}$. It extracts the "noise" components in the data and scales them (by scaling small singular values of inputs/features). Apparently this is a new augmentation operation, and its effectiveness has been experimentally verified on some datasets.
strength_and_weaknesses: **[Strength]**

- At a high level, this work has far-reaching implications for our community because: &nbsp; **(i)** Designing domain-independent data augmentation can facilitate the development of deep learning in different areas, considering that many data modalities **do not** have well-defined data augmentation operations (e.g., rotation, panning) as images do. &nbsp; **(ii)** We don't have many options for regularizing deep models on regression tasks (e.g., label smoothing can only be used for classification).
- The proposed operator ($\texttt{comfort-zone}$) is flexible because it can be applied in both input space and intermediate hidden space. Also, using SVD and scaling the "noise" or "trivial" components to generate new data is intuitive.
- $\texttt{Comfort-zone}$ yields better regularization effect than its counterpart (noise injection and mixup) on some benchmarks.

**[Weaknesses]**

- $\texttt{Comfort-zone}$ seems to be the opposite of noise injection, as it is trying to reduce the unimportant components in the data ($\lambda \in [0, 1]$). This seems to be contrary to the original purpose of data augmentation (to increase the noise in the data). What do the authors think about this?
- Performance improvement from $\texttt{comfort-zone}$ on large-scale datasets (in Sec. 5.2) seems marginal.


clarity,_quality,_novelty_and_reproducibility: The article is innovative enough, clearly presented, and of high quality. I believe there are enough details to reimplement the proposed operator in the paper.
summary_of_the_review: It is necessary to explore domain-independent data augmentation as a general-purpose regularization tool. Its value is further highlighted by the applicability to regression tasks.
The proposed augmentation ($\texttt{comfort-zone}$) is well-motivated and innovative, with the only drawback that it brings very marginal improvements on large datasets.

My preliminary recommendation is to accept the article.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this manuscript, the authors proposed a method named comfort zone which utilize the SVD to augment data for regression problems. To demonstrate the effectiveness of proposed approach, experiments across different datasets and network architecture are conducted.  
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[Updates after rebuttals]

Really appreciate the authors’ efforts for addressing my concerns. After discussion with other reviewers, I finalized my score as a marginal acceptance as the proposed comfort zone is flexible and very interesting for independent domain augmentation. I also have some advises for the authors as follows. 
Firstly, as replied by the authors, the proposed comfort zone has higher complexity than mixup and additive noise. Applying the SVD on large scale datasets would lead to long training time and the experimental results also show that the proposed approach can only achieve marginal improvement as pointed out by other reviewers. Secondly, I slightly agree with authors’ claim that other data augmentation methods on mentioned time series references are not applicable to the regression settings they consider. However, I would still suggest applying the proposed approach to time series regression tasks as they are very important in many real-world applications. Lastly, there is still room for improvement in terms of novelty and evaluation as mentioned by other reviewers. 



strength_and_weaknesses: Strength
+ the paper is clearly written and easy to follow. The proposed method is technically sound. 
+ Extensive experiments are conducted to verify the effectiveness of proposed method.

Weakness
- Lack of comparison with SOTA data augmentation approaches 
- Lack of comparison with other augmentation approaches in terms of complexity
clarity,_quality,_novelty_and_reproducibility: This manuscript is clearly written and easy to follow. The proposed method is novel for data augmentation in regression problems. The pseudocode provided is easy to reproduce the experiments
summary_of_the_review: Some issues need to be further clarified by the authors:

1. It is better provide the complexity comparison between the proposed one and other approaches.
2.  Besides the mix-up and UN, other advanced data augmentation approaches in regression problems should be compared. For instance, the permutation-and-jitter strategy in reference [1], decomposition-based methods [2,3] or generative models [4]. 

[1] Time-Series Representation Learning via Temporal and Contextual Contrasting
[2] Robusttad: Robust time series anomaly detection via decomposition and convolutional neural networks.
[3] Fast RobustSTL: Efficient and robust seasonal-trend decomposition for time series with complex patterns
[4] Time-series generative adversarial networks

3. The experiments of using proposed CZ in latent space is insufficient if the authors want to claim their approach suitable for latent space.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a data-driven method called comfort-zone for data augmentation of regression tasks. By producing new samples from the given ones by scaling their small singular values by random values, it incorporates the assumption that dominant components of the train set can also be viewed as true samples. Author[s] provide both non-differentiable input-level and differentiable pipeline which is applicable to any layer. Results on small and medium regression tasks and time-series forecasting show its effectiveness. 
strength_and_weaknesses: [Strength]

1. A nice idea of applying dominant components to mixup so that the new data points can mostly preserve the statistics of true samples.
2. Author[s] provide a comprehensive analysis of its relation to additive noise and VRMs.
3. Results on small/medium regression datasets are strong. 

[Weakness]

1. Could author[s] provide any insights on why it cannot be used as a drop-in replacement to standard mixup (for classification)?
2. Isn't comfort-zone encouraging stronger local manifolds around data points? Fig. 1 sounds like an overfit to the training data. How about the results used in domain adaptation, e.g. works like https://arxiv.org/abs/2001.00677? 


clarity,_quality,_novelty_and_reproducibility: This paper is well-written, with good motivation and comprehensive analysis, experimental results on small/medium regression tasks are strong, but not clear how well it can be for classification tasks as well as domain adaptation applications.  
summary_of_the_review: A simple drop-in approach for many data augmentation applications with comprehensive analysis. Results on regression tasks look good but need to see its performance on classification tasks and domain adaptation applications to verify its potential.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a domain-independent approach inspired by mixup, which they call comfort zone. Comfort zone works by taking the SVD of the matrix formed by the concatenation of batch inputs and outputs, and then generating augmented date by reducing the scale of some subset of the singular values (the scale and number of values are parameters).

They test this approach on regression and time series forecasting benchmarks and achieve good performance relative to the baselines they have chosen.
strength_and_weaknesses: Strenghts:
- The paper is easy to follow, and explains the method well.
- Compared to the baselines, in the settings the authors evaluate on, their method performs well.

Weaknesses:
- The empirical evaluation is somewhat short. For the forecasting experiments, only N-BEATS is used. Does the method also work for other datasets (the commonly used ETT, Electricity, Weather, ...), models (N-HITS, Transformer models, TCNs, LSTMs...)? What is the impact of the method on different approaches? More experiments would be needed to make the conclusions more credible.
- While the approach is ingenious, taking the SVD of a batch and making modifications to some of the singular values is part of the set of transforms certain fields will naturally apply to their input data as part of pre-processing (e.g. signal processing). There is in this sense actual, but limited novelty.
- The method claims to be domain independent but would not actually work (without modifications) for a non-regression task, contrary to Mixup.
clarity,_quality,_novelty_and_reproducibility: Clarity: Clear.
Quality: The paper lacks enough experiments, but apart from this important point, is good.
Originality: Marginal. 
summary_of_the_review: The paper shows that attenuating some of the singular values can result in improved performance on some specific datasets. More experiments would be needed as discussed above to make those conclusions stronger. The novelty is debatable: SVD is designed so that removing singular values is natural.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed a data-driven method called comfort-zone for data augmentation in regression tasks. In particular, the authors utilized SVD as pre-processing, and then generated new samples by scaling the small singular values derived by SVD with random values. Experiments on regression tasks and time series forecasting tasks were conducted to show the effectiveness of the proposed method. 

Strengths: 
1. The paper is clearly written. The proposed method is very easy to follow.
2. The proposed method is flexible as it can work on both input space and latent feature space.
3. The model performance on small and medium regression tasks is very promising.

Weaknesses: 
1. The key idea is to do data augmentation based on the SVD outputs. The novelty is thus not sufficient.
2. In addition to transformer/TCN, some strong baselines like Informer, ESTformer and Autoformer should be included.
3. The performance improvement on large-scale datasets is marginal.
4. The complexity of SVD is a concern for large-scale datasets.

justification_for_why_not_higher_score: The reviewers have common concerns about the novelty and evaluation of the paper.
Decision: Reject