pointer/keyboard: Unset the entire focus when the surface is destroyed

Otherwise, we'll end up with dangling parts in our resource list.