Update only necessary layers