Only update layers when necessary