Strictly prioritizing privileged instructions over lower-priority ones can lead to potential limitations and unintended consequences. One major limitation is the risk of over-refusal, where the model may refuse to follow valid instructions from lower-priority sources even when they do not conflict with higher-priority instructions. This can result in reduced functionality and user satisfaction, as the model may miss out on valuable input or fail to provide the expected responses.
To address these limitations, it is important to strike a balance between following privileged instructions and considering lower-priority instructions when they do not conflict. This can be achieved through a more nuanced training approach that teaches the model to differentiate between aligned and misaligned instructions, allowing it to follow lower-priority instructions when appropriate. Additionally, incorporating feedback mechanisms or reinforcement learning can help the model learn to adapt its decision-making process based on the specific context and requirements of each interaction.