pointer/keyboard: Fix segfault once more

If the resource is destroyed before the surface, then we'll kill
ourselves. Why can I never seem to write these correctly...