While DRO-based robust loss formulations offer several advantages, they also come with potential limitations and drawbacks:
Computational Complexity: Optimizing DRO-based losses can be computationally intensive, especially when dealing with large datasets or complex models, leading to longer training times.
Sensitivity to Hyperparameters: DRO formulations often involve tuning hyperparameters such as the temperature parameter and regularization terms, which can be challenging and require careful selection to achieve optimal performance.
Risk of Overfitting: DRO formulations may be prone to overfitting, especially when the model capacity is high or the dataset is small, leading to reduced generalization on unseen data.
Interpretability: DRO-based losses may be less interpretable compared to traditional loss functions, making it harder to understand the model's decision-making process.
Limited Scalability: DRO formulations may face scalability issues when applied to very large datasets or models, as the optimization process becomes more complex and resource-intensive.