Make magic parameter handling more robust

That way, we can fix the behavior on `remove()` (was a regression)