That is actually how I originally implemented it but found performance considerably worse for numeric fields.  This is due to the fact to get the `BytesRef` from the fielddata it does a `new BytesRef(Long.toString(val))`.  These string conversions killed performance to the point where the joins on numeric fields were 2x as slow as string field.  Now that I stick with the primitive types the joins on numeric fields are ~2x faster than string fields.  I figured the more complex code was worth the performance increase.