Repository: deepjavalibrary/djl
Pull request #1701 (2022-06-07 23:18:20)

Normalized Numeric Featurizer
Earlier, the AirfoilRandomAccess had to use custom code to add normalizing
functionality and couldn't rely on the featurizers. This creates a new
normalizing featurizer using the new PreparedFeaturizer and brings it into line
with the standard tabular dataset.

The changes in testAirfoilRemotePreprocessing come from an update to the
normalization. Before, it was normalized by calculating using the size (limited
to top 1500 elements in the dataset for the test). Instead, it normalizes using
the full dataset availableSize(). I feel like this would make more sense to not
have your data preprocessing be different between a limited dataset and the full
one and would be unexpected for users if it were.


codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1701?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
> Merging [#1701](https://codecov.io/gh/deepjavalibrary/djl/pull/1701?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (8fd3fbc) into [master](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bb5073f) will **decrease** coverage by `1.64%`.
> The diff coverage is `69.11%`.

```diff
@@             Coverage Diff              @@
##             master    #1701      +/-   ##
============================================
- Coverage     72.08%   70.44%   -1.65%     
- Complexity     5126     5510     +384     
============================================
  Files           473      522      +49     
  Lines         21970    24424    +2454     
  Branches       2351     2645     +294     
============================================
+ Hits          15838    17206    +1368     
- Misses         4925     5911     +986     
- Partials       1207     1307     +100     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/1701?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1701/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...rc/main/java/ai/djl/modality/cv/output/Joints.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1701/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9Kb2ludHMuamF2YQ==) | `71.42% <ø> (ø)` | |
| [.../main/java/ai/djl/modality/cv/output/Landmark.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1701/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9MYW5kbWFyay5qYXZh) | `100.00% <ø> (ø)` | |
| [...main/java/ai/djl/modality/cv/output/Rectangle.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1701/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9SZWN0YW5nbGUuamF2YQ==) | `72.41% <0.00%> (ø)` | |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1701/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <ø> (-5.24%)` | :arrow_down: |
| [...odality/cv/translator/BigGANTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1701/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `33.33% <0.00%> (+8.33%)` | :arrow_up: |
| [...nslator/InstanceSegmentationTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1701/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW5zdGFuY2VTZWdtZW50YXRpb25UcmFuc2xhdG9yRmFjdG9yeS5qYXZh) | `14.28% <0.00%> (-3.90%)` | :arrow_down: |
| [.../ai/djl/modality/cv/translator/YoloTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1701/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1RyYW5zbGF0b3IuamF2YQ==) | `8.33% <0.00%> (-0.50%)` | :arrow_down: |
| [.../modality/cv/translator/YoloTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1701/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1RyYW5zbGF0b3JGYWN0b3J5LmphdmE=) | `8.33% <0.00%> (-1.67%)` | :arrow_down: |
| [...i/djl/modality/cv/translator/YoloV5Translator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1701/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1Y1VHJhbnNsYXRvci5qYXZh) | `5.38% <0.00%> (-0.31%)` | :arrow_down: |
| ... and [282 more](https://codecov.io/gh/deepjavalibrary/djl/pull/1701/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1701?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1701?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Last update [d1e2b5f...8fd3fbc](https://codecov.io/gh/deepjavalibrary/djl/pull/1701?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/basicdataset/src/main/java/ai/djl/basicdataset/tabular/AirfoilRandomAccess.java b/basicdataset/src/main/java/ai/djl/basicdataset/tabular/AirfoilRandomAccess.java
index d100d3516..4b68eb5a3 100644
--- a/basicdataset/src/main/java/ai/djl/basicdataset/tabular/AirfoilRandomAccess.java
+++ b/basicdataset/src/main/java/ai/djl/basicdataset/tabular/AirfoilRandomAccess.java
@@ -15,23 +15,15 @@ package ai.djl.basicdataset.tabular;
 import ai.djl.Application.Tabular;
 import ai.djl.basicdataset.BasicDatasets;
 import ai.djl.basicdataset.tabular.utils.Feature;
-import ai.djl.ndarray.NDList;
-import ai.djl.ndarray.NDManager;
-import ai.djl.ndarray.types.Shape;
 import ai.djl.repository.Artifact;
 import ai.djl.repository.MRL;
 import ai.djl.repository.Repository;
 import ai.djl.util.Progress;
 import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.nio.FloatBuffer;
 import java.nio.file.Path;
 import java.util.Arrays;
-import java.util.HashMap;
 import java.util.List;
-import java.util.Map;
 import org.apache.commons.csv.CSVFormat;
-import org.apache.commons.csv.CSVRecord;
 
 /**
  * Airfoil Self-Noise Data Set from <a
@@ -52,10 +44,6 @@ public final class AirfoilRandomAccess extends CsvDataset {
     private Usage usage;
     private boolean prepared;
 
-    private boolean normalize;
-    private Map<String, Float> mean;
-    private Map<String, Float> std;
-
     /**
      * Creates an instance of {@code RandomAccessDataset} with the arguments in {@link Builder}.
      *
@@ -65,7 +53,6 @@ public final class AirfoilRandomAccess extends CsvDataset {
         super(builder);
         usage = builder.usage;
         mrl = builder.getMrl();
-        normalize = builder.normalize;
     }
 
     /** {@inheritDoc} */
@@ -93,20 +80,6 @@ public final class AirfoilRandomAccess extends CsvDataset {
 
         csvUrl = csvFile.toUri().toURL();
         super.prepare(progress);
-
-        if (normalize) {
-            mean = new HashMap<>();
-            std = new HashMap<>();
-
-            for (Feature feature : features) {
-                calculateMean(feature.getName());
-                calculateStd(feature.getName());
-            }
-            for (Feature feature : labels) {
-                calculateMean(feature.getName());
-                calculateStd(feature.getName());
-            }
-        }
         prepared = true;
     }
 
@@ -116,46 +89,6 @@ public final class AirfoilRandomAccess extends CsvDataset {
         return Arrays.asList(COLUMNS).subList(0, 5);
     }
 
-    /** {@inheritDoc} */
-    @Override
-    public NDList getRowFeatures(NDManager manager, long index, List<Feature> selected) {
-        CSVRecord record = csvRecords.get(Math.toIntExact(index));
-        int length = selected.size();
-        ByteBuffer bb = manager.allocateDirect(length * 4);
-        FloatBuffer buf = bb.asFloatBuffer();
-        for (Feature feature : selected) {
-            String name = feature.getName();
-            float value = Float.parseFloat(record.get(name));
-            if (normalize) {
-                value = (value - mean.get(name)) / std.get(name);
-            }
-            buf.put(value);
-        }
-        buf.rewind();
-        return new NDList(manager.create(buf, new Shape(length)));
-    }
-
-    private void calculateMean(String column) {
-        float sum = 0;
-        long size = size();
-        for (int i = 0; i < size; ++i) {
-            CSVRecord record = csvRecords.get(i);
-            sum += Float.parseFloat(record.get(column));
-        }
-        mean.put(column, sum / size);
-    }
-
-    private void calculateStd(String column) {
-        float average = mean.get(column);
-        float sum = 0;
-        long size = size();
-        for (int i = 0; i < size; ++i) {
-            CSVRecord record = csvRecords.get(i);
-            sum += (float) Math.pow(Float.parseFloat(record.get(column)) - average, 2);
-        }
-        std.put(column, (float) Math.sqrt(sum / size));
-    }
-
     /**
      * Creates a builder to build a {@link AirfoilRandomAccess}.
      *
@@ -280,11 +213,11 @@ public final class AirfoilRandomAccess extends CsvDataset {
         public AirfoilRandomAccess build() {
             if (features.isEmpty()) {
                 for (int i = 0; i < 5; ++i) {
-                    addFeature(COLUMNS[i]);
+                    addNumericFeature(COLUMNS[i], normalize);
                 }
             }
             if (labels.isEmpty()) {
-                addNumericLabel("ssoundpres");
+                addNumericLabel("ssoundpres", normalize);
             }
             return new AirfoilRandomAccess(this);
         }
diff --git a/basicdataset/src/main/java/ai/djl/basicdataset/tabular/AmesRandomAccess.java b/basicdataset/src/main/java/ai/djl/basicdataset/tabular/AmesRandomAccess.java
index e8a5b3694..538c0f95e 100644
--- a/basicdataset/src/main/java/ai/djl/basicdataset/tabular/AmesRandomAccess.java
+++ b/basicdataset/src/main/java/ai/djl/basicdataset/tabular/AmesRandomAccess.java
@@ -200,7 +200,7 @@ public class AmesRandomAccess extends CsvDataset {
             if (af.categorical.contains(name)) {
                 Map<String, Integer> map = af.featureToMap.get(name);
                 if (map == null) {
-                    return addCategoricalFeature(name);
+                    return addCategoricalFeature(name, onehotEncode);
                 }
                 return addCategoricalFeature(name, map, onehotEncode);
             }
diff --git a/basicdataset/src/main/java/ai/djl/basicdataset/tabular/TabularDataset.java b/basicdataset/src/main/java/ai/djl/basicdataset/tabular/TabularDataset.java
index 08adebe39..114b0033d 100644
--- a/basicdataset/src/main/java/ai/djl/basicdataset/tabular/TabularDataset.java
+++ b/basicdataset/src/main/java/ai/djl/basicdataset/tabular/TabularDataset.java
@@ -14,6 +14,7 @@ package ai.djl.basicdataset.tabular;
 
 import ai.djl.basicdataset.tabular.utils.DynamicBuffer;
 import ai.djl.basicdataset.tabular.utils.Feature;
+import ai.djl.basicdataset.tabular.utils.Featurizers;
 import ai.djl.basicdataset.tabular.utils.PreparedFeaturizer;
 import ai.djl.ndarray.NDList;
 import ai.djl.ndarray.NDManager;
@@ -87,7 +88,7 @@ public abstract class TabularDataset extends RandomAccessDataset {
         for (Feature feature : featuresToPrepare) {
             if (feature.getFeaturizer() instanceof PreparedFeaturizer) {
                 PreparedFeaturizer featurizer = (PreparedFeaturizer) feature.getFeaturizer();
-                List<String> inputs = new ArrayList<>(Math.toIntExact(availableSize()));
+                List<String> inputs = new ArrayList<>(Math.toIntExact(availableSize));
                 for (int i = 0; i < availableSize; i++) {
                     inputs.add(getCell(i, feature.getName()));
                 }
@@ -143,6 +144,18 @@ public abstract class TabularDataset extends RandomAccessDataset {
             return self();
         }
 
+        /**
+         * Adds a numeric feature to the feature set.
+         *
+         * @param name the feature name
+         * @param normalize true to normalize the column
+         * @return this builder
+         */
+        public T addNumericFeature(String name, boolean normalize) {
+            features.add(new Feature(name, Featurizers.getNumericFeaturizer(normalize)));
+            return self();
+        }
+
         /**
          * Adds a categorical feature to the feature set.
          *
@@ -154,12 +167,24 @@ public abstract class TabularDataset extends RandomAccessDataset {
             return self();
         }
 
+        /**
+         * Adds a categorical feature to the feature set.
+         *
+         * @param name the feature name
+         * @param onehotEncode true to use onehot encode
+         * @return this builder
+         */
+        public T addCategoricalFeature(String name, boolean onehotEncode) {
+            features.add(new Feature(name, Featurizers.getStringFeaturizer(onehotEncode)));
+            return self();
+        }
+
         /**
          * Adds a categorical feature to the feature set with specified mapping.
          *
          * @param name the feature name
          * @param map a map contains categorical value maps to index
-         * @param onehotEncode true if use onehot encode
+         * @param onehotEncode true to use onehot encode
          * @return this builder
          */
         public T addCategoricalFeature(
@@ -190,6 +215,18 @@ public abstract class TabularDataset extends RandomAccessDataset {
             return self();
         }
 
+        /**
+         * Adds a number feature to the label set.
+         *
+         * @param name the label name
+         * @param normalize true to normalize the column
+         * @return this builder
+         */
+        public T addNumericLabel(String name, boolean normalize) {
+            labels.add(new Feature(name, Featurizers.getNumericFeaturizer(normalize)));
+            return self();
+        }
+
         /**
          * Adds a categorical feature to the label set.
          *
@@ -197,7 +234,19 @@ public abstract class TabularDataset extends RandomAccessDataset {
          * @return this builder
          */
         public T addCategoricalLabel(String name) {
-            labels.add(new Feature(name, true));
+            labels.add(new Feature(name, false));
+            return self();
+        }
+
+        /**
+         * Adds a categorical feature to the label set.
+         *
+         * @param name the feature name
+         * @param onehotEncode true if use onehot encode
+         * @return this builder
+         */
+        public T addCategoricalLabel(String name, boolean onehotEncode) {
+            labels.add(new Feature(name, Featurizers.getStringFeaturizer(onehotEncode)));
             return self();
         }
 
diff --git a/basicdataset/src/main/java/ai/djl/basicdataset/tabular/utils/Featurizers.java b/basicdataset/src/main/java/ai/djl/basicdataset/tabular/utils/Featurizers.java
index c41490850..26a20d0d2 100644
--- a/basicdataset/src/main/java/ai/djl/basicdataset/tabular/utils/Featurizers.java
+++ b/basicdataset/src/main/java/ai/djl/basicdataset/tabular/utils/Featurizers.java
@@ -23,7 +23,6 @@ import java.util.concurrent.ConcurrentHashMap;
 public final class Featurizers {
 
     private static final Featurizer NUMERIC_FEATURIZER = new NumericFeaturizer();
-    private static final Featurizer STRING_FEATURIZER = new StringFeaturizer();
 
     private Featurizers() {}
 
@@ -33,7 +32,21 @@ public final class Featurizers {
      * @return the default numeric {@link Featurizer}
      */
     public static Featurizer getNumericFeaturizer() {
-        return NUMERIC_FEATURIZER;
+        return getNumericFeaturizer(false);
+    }
+
+    /**
+     * Returns the default numeric {@link Featurizer}.
+     *
+     * @param normalize true to normalize (with mean and std) the values
+     * @return the default numeric {@link Featurizer}
+     */
+    public static Featurizer getNumericFeaturizer(boolean normalize) {
+        if (normalize) {
+            return new NormalizedNumericFeaturizer();
+        } else {
+            return NUMERIC_FEATURIZER;
+        }
     }
 
     /**
@@ -55,7 +68,7 @@ public final class Featurizers {
         if (onehotEncode) {
             return new PreparedOneHotStringFeaturizer();
         } else {
-            return STRING_FEATURIZER;
+            return new StringFeaturizer();
         }
     }
 
@@ -94,6 +107,42 @@ public final class Featurizers {
         }
     }
 
+    private static final class NormalizedNumericFeaturizer implements PreparedFeaturizer {
+
+        private float mean;
+        private float std;
+
+        /** {@inheritDoc} */
+        @Override
+        public void featurize(DynamicBuffer buf, String input) {
+            float value = (Float.parseFloat(input) - mean) / std;
+            buf.put(value);
+        }
+
+        /** {@inheritDoc} */
+        @Override
+        public void prepare(List<String> inputs) {
+            calculateMean(inputs);
+            calculateStd(inputs);
+        }
+
+        private void calculateMean(List<String> inputs) {
+            double sum = 0;
+            for (String input : inputs) {
+                sum += Float.parseFloat(input);
+            }
+            mean = (float) (sum / inputs.size());
+        }
+
+        private void calculateStd(List<String> inputs) {
+            double sum = 0;
+            for (String input : inputs) {
+                sum += Math.pow(Float.parseFloat(input) - mean, 2);
+            }
+            std = (float) Math.sqrt(sum / inputs.size());
+        }
+    }
+
     private static class OneHotStringFeaturizer implements Featurizer {
         protected Map<String, Integer> map;
 


Test patch:
diff --git a/basicdataset/src/test/java/ai/djl/basicdataset/AirfoilRandomAccessTest.java b/basicdataset/src/test/java/ai/djl/basicdataset/AirfoilRandomAccessTest.java
index dc8028c5c..63ebbbbc1 100644
--- a/basicdataset/src/test/java/ai/djl/basicdataset/AirfoilRandomAccessTest.java
+++ b/basicdataset/src/test/java/ai/djl/basicdataset/AirfoilRandomAccessTest.java
@@ -100,8 +100,8 @@ public class AirfoilRandomAccessTest {
 
         float epsilon = (float) 1e-4;
 
-        float[] expected = {-0.6603f, -1.1448f, 1.797f, 1.3109f, -0.6443f};
+        float[] expected = {-0.6620f, -1.1464f, 1.7993f, 1.3129f, -0.6448f};
         Assert.assertEquals(data.head().toFloatArray(), expected, epsilon);
-        Assert.assertEquals(labels.head().toFloatArray(), new float[] {0.1937f}, epsilon);
+        Assert.assertEquals(labels.head().toFloatArray(), new float[] {0.1979f}, epsilon);
     }
 }