do not set colors once again at every buffer switch

Now colors only get set at plugin loading and when :colo is used.