Proper handling for removing user meta

Fix issue #2630