how is the remove case different from removexattr?

It sounds like removexattr(..., foo) is the same as setxattr(..., foo, NULL, 0).  If so, wouldn't that translation happen on the client side?