Don't include metadata if empty in Op.to_primitives