Combining Natural-Instruct (NI) and Self-Instruct (SI) data has some potential drawbacks:
Data Inconsistencies: NI data may contain improper coding formats or ambiguous variable names, which could introduce noise when combined with SI data that is properly formatted.
Diversity vs Quality Trade-off: While combining both datasets increases diversity, there might be a trade-off between diversity and quality. The models trained on this combined dataset may struggle with balancing these aspects effectively.
Model Bias: Models trained on combined NI-SI datasets may exhibit biases towards certain types of instructions or coding styles present in one dataset over another.