Repository: pandas-dev/pandas
Pull request: ENH: Enable read_csv interpret 'Infinity' as floating point value #10065

- [x] closes #10065
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
Enable read_csv to interpret "Infinity", "+Infinity" and "-Infinity" as floating point values
The strings "Infinity", "+Infinity" and "-Infinity" are commonly used to represent infinite floating point values. It would be great for pandas pd.read_csv to correctly interpret these strings as infinite floating point values or to take a parameter "inf_values" analogous to "na_values" that would allow the caller to specify what they require.

Hints from issue discussion:
`inf/-inf` case insensitive are already accepted. These are standard. 

We're reading CSVs produced by Java, which stringifies infinite values as "Infinity" and "-Infinity".

Am I be looking in the right place if I'm poking around in [maybe_convert_numeric()](https://github.com/pydata/pandas/blob/master/pandas/src/inference.pyx#L505).

Here is where it would need to be changed: https://github.com/pydata/pandas/blob/master/pandas/parser.pyx#L1474

I think you might just be able to short-circuit with a `strncasecmp` (to only compare first n characters), then to compare versus a hash table of allowed values. These c-functions are used because this is a very perf intensive part of the code (iow its hit a lot), so would need some testing to make sure perf is ok.

Commit message: Fix formatting

Patch to review:
diff --git a/doc/source/whatsnew/v0.25.2.rst b/doc/source/whatsnew/v0.25.2.rst
index 6974c7521a237..8f4b278a014a9 100644
--- a/doc/source/whatsnew/v0.25.2.rst
+++ b/doc/source/whatsnew/v0.25.2.rst
@@ -61,7 +61,7 @@ Missing
 
 I/O
 ^^^
-
+- Improve infinity parsing. ``pd.read_csv()`` will now interpret ``Infinity``, ``+Infinity``, ``-Infinity`` as floating point values  (:issue:`10065`).
 -
 -
 -
diff --git a/pandas/_libs/parsers.pyx b/pandas/_libs/parsers.pyx
index 6cc9dd22ce7c9..62a3568932def 100644
--- a/pandas/_libs/parsers.pyx
+++ b/pandas/_libs/parsers.pyx
@@ -1693,6 +1693,10 @@ cdef:
     char* cposinf = b'+inf'
     char* cneginf = b'-inf'
 
+    char* cinfty = b'Infinity'
+    char* cposinfty = b'+Infinity'
+    char* cneginfty = b'-Infinity'
+
 
 cdef _try_double(parser_t *parser, int64_t col,
                  int64_t line_start, int64_t line_end,
@@ -1772,9 +1776,12 @@ cdef inline int _try_double_nogil(parser_t *parser,
                 if error != 0 or p_end == word or p_end[0]:
                     error = 0
                     if (strcasecmp(word, cinf) == 0 or
-                            strcasecmp(word, cposinf) == 0):
+                            strcasecmp(word, cposinf) == 0 or
+                            strcasecmp(word, cinfty) == 0 or
+                            strcasecmp(word, cposinfty) == 0):
                         data[0] = INF
-                    elif strcasecmp(word, cneginf) == 0:
+                    elif (strcasecmp(word, cneginf) == 0 or
+                            strcasecmp(word, cneginfty) == 0 ):
                         data[0] = NEGINF
                     else:
                         return 1
@@ -1793,9 +1800,12 @@ cdef inline int _try_double_nogil(parser_t *parser,
             if error != 0 or p_end == word or p_end[0]:
                 error = 0
                 if (strcasecmp(word, cinf) == 0 or
-                        strcasecmp(word, cposinf) == 0):
+                        strcasecmp(word, cposinf) == 0 or
+                        strcasecmp(word, cinfty) == 0 or
+                        strcasecmp(word, cposinfty) == 0):
                     data[0] = INF
-                elif strcasecmp(word, cneginf) == 0:
+                elif (strcasecmp(word, cneginf) == 0 or
+                        strcasecmp(word, cneginfty) == 0):
                     data[0] = NEGINF
                 else:
                     return 1
diff --git a/pandas/_libs/src/parse_helper.h b/pandas/_libs/src/parse_helper.h
index 1db1878a8a773..1db4c813bb493 100644
--- a/pandas/_libs/src/parse_helper.h
+++ b/pandas/_libs/src/parse_helper.h
@@ -50,7 +50,7 @@ int floatify(PyObject *str, double *result, int *maybe_int) {
     status = to_double(data, result, sci, dec, maybe_int);
 
     if (!status) {
-        /* handle inf/-inf */
+        /* handle inf/-inf infinity/-infinity */
         if (strlen(data) == 3) {
             if (0 == strcasecmp(data, "inf")) {
                 *result = HUGE_VAL;
@@ -68,6 +68,23 @@ int floatify(PyObject *str, double *result, int *maybe_int) {
             } else {
                 goto parsingerror;
             }
+        } else if (strlen(data) == 8) {
+            if (0 == strcasecmp(data, "infinity")) {
+                *result = HUGE_VAL;
+                *maybe_int = 0;
+            } else {
+                goto parsingerror;
+            }
+        } else if (strlen(data) == 9) {
+            if (0 == strcasecmp(data, "-infinity")) {
+                *result = -HUGE_VAL;
+                *maybe_int = 0;
+            } else if (0 == strcasecmp(data, "+infinity")) {
+                *result = HUGE_VAL;
+                *maybe_int = 0;
+            } else {
+                goto parsingerror;
+            }
         } else {
             goto parsingerror;
         }
diff --git a/pandas/tests/io/parser/test_common.py b/pandas/tests/io/parser/test_common.py
index e04535df56663..a3473e634c468 100644
--- a/pandas/tests/io/parser/test_common.py
+++ b/pandas/tests/io/parser/test_common.py
@@ -1865,6 +1865,22 @@ def test_inf_parsing(all_parsers, na_filter):
     tm.assert_frame_equal(result, expected)
 
 
+@pytest.mark.parametrize("na_filter", [True, False])
+def test_infinity_parsing(all_parsers, na_filter):
+    parser = all_parsers
+    data = """\
+,A
+a,Infinity
+b,-Infinity
+c,+Infinity
+"""
+    expected = DataFrame(
+        {"A": [float("inf"), float("-inf"), float("inf")]}, index=["a", "b", "c"]
+    )
+    result = parser.read_csv(StringIO(data), index_col=0, na_filter=na_filter)
+    tm.assert_frame_equal(result, expected)
+
+
 @pytest.mark.parametrize("nrows", [0, 1, 2, 3, 4, 5])
 def test_raise_on_no_columns(all_parsers, nrows):
     parser = all_parsers


Code review comments:
diff_hunk: @@ -61,7 +61,7 @@ Missing
 
 I/O
 ^^^
-
+- Improve infinity parsing. ``pd.read_csv()`` will now interpret ``Infinity``, ``+Infinity``, ``-Infinity`` as floating point values  (:issue:`10065`).
path: doc/source/whatsnew/v0.25.2.rst
text: @user1:
Can you move this to 1.0.0

@author:
Done. Also made test case more clear comparing with `float("infinity")`.
BTW not only Fortran prints infinities like that. Checkout this Java snippet
https://onlinegdb.com/r1GTsZPBB
```Java
public class Main{
     public static void main(String []args){
        System.out.println(Double.POSITIVE_INFINITY);
        System.out.println(Double.NEGATIVE_INFINITY);
     }
}
```
 https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Double.html#toString(double)
If m is infinity, it is represented by the characters "Infinity"; thus, positive infinity produces the result "Infinity" and negative infinity produces the result "-Infinity".