Repository: jtablesaw/tablesaw
Pull request #906 (2021-04-25 07:43:35)

Support percentage by parsing percentage to double
Thanks for contributing.

- [x] Tick to sign-off your agreement to the [Developer Certificate of Origin (DCO) 1.1](https://developercertificate.org)

## Description

What was changed

`canParse()` and `parseDouble()` in *DoubleParser*
They are modified to accept the strings that contains number and end with %.
Thecorressponding issue is #204 .

## Testing

Did you add a unit test?

Yes. In *CsvReaderTest* and *DoubleParserTest*


lwhite1: @benmccann do you happen to know if the use of an appropriate locale in ReadOptions would permit reading numbers with "," as a decimal point? 
If not, that would also require number parsing changes.  If there are other common use-cases, it makes me wonder if this isn't too specialized to add as its own option. Thoughts?
benmccann: I'm not sure what using locale for "," as decimal separator and then handling percentages have to do with one another. Maybe I'm not understanding?
lwhite1: I think they're both cases typically handled in Java by DecimalFormat or NumberFormat, so any solution to the "," issue that used a DecimalFormat, would probably also solve the percentage issue.
benmccann: Ah, okay. I'm not sure if that's supported yet, but it sounds like a good solution to this and would be interesting to test. The majority of European countries use a comma for decimal separator such as Spain, France, Norway, and Denmark.
lwhite1: I checked the code and Locale is only accessed by Date/Time column parsers. I will try do a little research on number parsing with locales. 

Looking at the changes proposed here, the hard-coded reference to `AbstractColumnParser.remove(s, ',')` seems like a locale-related issue in the making. Unless, I'm missing something, I don't think it should be approved as-is. 
lwhite1: I'm still not ready to approve this. It continues to seem too limited and specialized to get it's own option.
I checked both Pandas and R, neither provide a special hook for percentages. 

- Pandas provides an option for supplying a map of column-names to conversion functions, which it calls 'converters'
- R relies on post-import processing: using a function to convert the String column to a numeric column after the data is loaded, which as @benmccann pointed out elsewhere, we already support using the mapInto() method. 

The Pandas solution is nice and general, it can be used by any columnType to convert the data to any other type. I'm not completely sure we can do that given the added complexity of strong types and generics, but it seems like we should try - if the problem is indeed worth solving.  
lwhite1: @benmccann I'm going to close this (and the other, older version) on the grounds of method proliferation (yes i know i'm not real consistent on that). 

What are your thoughts about making this the default behavior by modifying the column type detection for floats and doubles to consider percentages?
benmccann: I'd be fine with that
lwhite1: @Kerwinooooo If you resolve the conflicts I will merge this.  Thank you.
emilianbold: As long as you made a decision about percentages do comment/close on #703 too please.
Kerwinooooo: > @Kerwinooooo If you resolve the conflicts I will merge this. Thank you.

Conflict is resolved. Thank you.

Patch:
diff --git a/core/src/main/java/tech/tablesaw/columns/numbers/DoubleParser.java b/core/src/main/java/tech/tablesaw/columns/numbers/DoubleParser.java
index f6ac951a..4eb74b3d 100644
--- a/core/src/main/java/tech/tablesaw/columns/numbers/DoubleParser.java
+++ b/core/src/main/java/tech/tablesaw/columns/numbers/DoubleParser.java
@@ -1,6 +1,8 @@
 package tech.tablesaw.columns.numbers;
 
 import com.google.common.collect.Lists;
+import java.text.NumberFormat;
+import java.text.ParseException;
 import tech.tablesaw.api.ColumnType;
 import tech.tablesaw.columns.AbstractColumnParser;
 import tech.tablesaw.io.ReadOptions;
@@ -24,9 +26,14 @@ public class DoubleParser extends AbstractColumnParser<Double> {
       return true;
     }
     try {
-      Double.parseDouble(AbstractColumnParser.remove(s, ','));
+      if (isPercent(AbstractColumnParser.remove(s, ','))) {
+        s = AbstractColumnParser.remove(s, ',');
+        Number number = NumberFormat.getPercentInstance().parse(s);
+      } else {
+        Double.parseDouble(AbstractColumnParser.remove(s, ','));
+      }
       return true;
-    } catch (NumberFormatException e) {
+    } catch (NumberFormatException | ParseException e) {
       // it's all part of the plan
       return false;
     }
@@ -42,6 +49,14 @@ public class DoubleParser extends AbstractColumnParser<Double> {
     if (isMissing(s)) {
       return DoubleColumnType.missingValueIndicator();
     }
+    if (isPercent(AbstractColumnParser.remove(s, ','))) {
+      s = AbstractColumnParser.remove(s, ',').substring(0, s.length() - 1);
+      return Double.parseDouble(s) / 100.0;
+    }
     return Double.parseDouble(AbstractColumnParser.remove(s, ','));
   }
+
+  private boolean isPercent(String s) {
+    return s.charAt(s.length() - 1) == '%';
+  }
 }
diff --git a/data/currency_percent.csv b/data/currency_percent.csv
new file mode 100644
index 00000000..ceddacb5
--- /dev/null
+++ b/data/currency_percent.csv
@@ -0,0 +1,4 @@
+Currency,Percentage,double
+$10.3,1.32%,13.14
+$50.10,32.768%,35
+$10,100%,93.131
\ No newline at end of file


Test patch:
diff --git a/core/src/test/java/tech/tablesaw/api/DoubleParserTest.java b/core/src/test/java/tech/tablesaw/api/DoubleParserTest.java
new file mode 100644
index 00000000..e6032a28
--- /dev/null
+++ b/core/src/test/java/tech/tablesaw/api/DoubleParserTest.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package tech.tablesaw.api;
+
+import static org.junit.jupiter.api.Assertions.*;
+
+import org.junit.jupiter.api.Test;
+import tech.tablesaw.columns.numbers.DoubleParser;
+
+class DoubleParserTest {
+  @Test
+  void testCanParse1() {
+    DoubleParser doubleParser = new DoubleParser(ColumnType.DOUBLE);
+    assertTrue(doubleParser.canParse("1.3"));
+  }
+
+  @Test
+  void testCanParse2() {
+    DoubleParser doubleParser = new DoubleParser(ColumnType.DOUBLE);
+    assertTrue(doubleParser.canParse("1.3%"));
+  }
+
+  @Test
+  void testCanParse3() {
+    DoubleParser doubleParser = new DoubleParser(ColumnType.DOUBLE);
+    assertFalse(doubleParser.canParse("%"));
+  }
+
+  @Test
+  void testParseDouble1() {
+    DoubleParser doubleParser = new DoubleParser(ColumnType.DOUBLE);
+    assertEquals(1.3, doubleParser.parseDouble("1.3"));
+  }
+
+  @Test
+  void testParseDouble2() {
+    DoubleParser doubleParser = new DoubleParser(ColumnType.DOUBLE);
+    assertEquals(0.012, doubleParser.parseDouble("1.2%"));
+  }
+}
diff --git a/core/src/test/java/tech/tablesaw/io/csv/CsvReaderTest.java b/core/src/test/java/tech/tablesaw/io/csv/CsvReaderTest.java
index 4327f4fa..a4bef783 100644
--- a/core/src/test/java/tech/tablesaw/io/csv/CsvReaderTest.java
+++ b/core/src/test/java/tech/tablesaw/io/csv/CsvReaderTest.java
@@ -49,10 +49,13 @@ import org.junit.jupiter.api.Test;
 import tech.tablesaw.api.ColumnType;
 import tech.tablesaw.api.DateColumn;
 import tech.tablesaw.api.DateTimeColumn;
+import tech.tablesaw.api.DoubleColumn;
 import tech.tablesaw.api.LongColumn;
 import tech.tablesaw.api.ShortColumn;
 import tech.tablesaw.api.StringColumn;
 import tech.tablesaw.api.Table;
+import tech.tablesaw.columns.numbers.DoubleColumnType;
+import tech.tablesaw.columns.numbers.NumberColumnFormatter;
 import tech.tablesaw.io.AddCellToColumnException;
 
 /** Tests for CSV Reading */
@@ -846,6 +849,26 @@ public class CsvReaderTest {
     assertEquals(table.get(0, 0), out.get(0, 0));
   }
 
+  @Test
+  public void testReadCsvWithPercentage1() throws IOException {
+    Table table = Table.read().csv(CsvReadOptions.builder("../data/currency_percent.csv"));
+    assertEquals(DoubleColumnType.instance(), table.columnTypes()[1]);
+    assertEquals(DoubleColumnType.instance(), table.columnTypes()[2]);
+  }
+
+  @Test
+  public void testReadCsvWithPercentage2() throws IOException {
+    Table table = Table.read().csv(CsvReadOptions.builder("../data/currency_percent.csv"));
+    DoubleColumn column = (DoubleColumn) table.column(1);
+    assertEquals("0.0132", column.getString(0));
+    assertEquals("0.32768", column.getString(1));
+    assertEquals("1", column.getString(2));
+    column.setPrintFormatter(NumberColumnFormatter.percent(2));
+    assertEquals("1.32%", column.getString(0));
+    assertEquals("32.77%", column.getString(1));
+    assertEquals("100.00%", column.getString(2));
+  }
+
   @Test
   public void testSkipRowsWithInvalidColumnCount() throws IOException {
     Table table =