Inadequate power would be the primary reason for an unstable overclock. So when a GPU requires more power but can not get it, it leads to unstable overclocking.