While relying on pre-extracted Deep Support Vectors (DSVs) in scenarios with restricted model access offers significant advantages, there are potential limitations and drawbacks to consider:
Limited Adaptability: Pre-extracted DSVs may not adapt well to the specific characteristics of the dataset being distilled. Since DSVs are extracted from a pretrained model, they may not capture the nuances and intricacies of the new dataset, leading to suboptimal distillation performance.
Overfitting Concerns: DSVs extracted from a pretrained model may carry biases or overfitting tendencies from the original training data. When used in dataset distillation, these biases can affect the generalization ability of the distilled dataset, especially in scenarios with limited practical data.
Model Dependency: Relying solely on DSVs for distillation limits the flexibility and robustness of the method. If the pretrained model used to extract DSVs is not well-suited for the new dataset, the distilled dataset may not accurately represent the underlying data distribution.
Information Loss: Depending solely on DSVs for model knowledge may result in information loss, especially in scenarios where the DSVs do not adequately capture the diversity and complexity of the dataset. This can lead to a lack of representativeness in the distilled dataset.
Considering these limitations, it is essential to carefully balance the use of pre-extracted DSVs with other distillation techniques to ensure comprehensive and accurate dataset condensation in scenarios with restricted model access.