I don't think you should do this to be honest. ScriptDocValues are only for scripts. I think you should specialized this for field data and just do this:

* make `AtomicFieldData` an `AtomicNumericFieldData` <== this should only work on numerics

* don't pull script values:

```Java
 public void setNextReader(AtomicReaderContext context) {
      this.values = this.indexFieldData.load(context).doubleValues();
 }
public double score(int docId, float subQueryScore) {
  final int numValues = values.setDocId(docId);
  for (int i = 0; i < numValues; i++) {
         double val = values.nextValue();
         return Modifier.apply(modifier, val * boostFactor);
   }
   return 1.0d; // here return a missing value - the branch is too expensive in such a tight loop.
}
```
hope this makes sense?