ENH: Enable read_csv interpret 'Infinity' as floating point value #10065
State: closed


- [x] closes #10065
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
We should be careful about changing these. The last time we changed things we got a decent about of pushback about unintended behavior changes.
Yea I'd also be hesitant to do something like this. Code looks nice but given this isn't really standardized I would hate to start interpreting things that would really just be the string "Infinity" as a float value
@WillAyd, when you say 
> this isn't really standardized

Do you mean there is no specification (e.g. IEEE754) for this? De-facto, it is quite common to treat "Infinity" as infinite value
- Python does it 
```Python
assert float("Infinity") == float("inf")
``` 
The same thing  why this pull-request https://github.com/pandas-dev/pandas/pull/13274 was accepted.
- Numpy has similar function `numpy.genfromtxt` that interprets "Infinity" as inf
```Python
In [1]: import numpy as np                                                                                            
In [2]: from io import StringIO                                                                                       
In [3]: data =''' 
   ...: Infinity 
   ...: +Infinity 
   ...: -Infinity'''                                                                                                  
In [4]: np.genfromtxt(StringIO(data))                                                                                 
Out[4]: array([ inf,  inf, -inf])
```
- Fortran writes ∞ like that in formatted output (tested with GNU Fortran and Intel Fortran). 
This might be the reason for this [issue](https://github.com/PolyChord/PolyChordLite/issues/16).

- Java does the same https://onlinegdb.com/H12ipZwHB
```Java
public class Main{
     public static void main(String []args){
        System.out.println(Double.POSITIVE_INFINITY);
        System.out.println(Double.NEGATIVE_INFINITY);
     }
}
```
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Double.html#toString(double)
If m is infinity, it is represented by the characters "Infinity"; thus, positive infinity produces the result "Infinity" and negative infinity produces the result "-Infinity".
> I would hate to start interpreting things that would really just be the string "Infinity" as a float value

Ok, let it be string by default, as it is now. But what if I expect floating point values?
```python
pd.read_csv(fname, dtype=float)
``` 
This throws a TypeError and ValueError. Could we make it work in this case?

I realize, my fix might affect performance. Maybe,  there is a better solution. In https://github.com/pandas-dev/pandas/issues/10065#issuecomment-99270957 @jreback  suggested

>to short-circuit with a strncasecmp (to only compare first n characters), then to compare versus a hash table of allowed values.

Should  hash table be a local variable in [_try_double function]( https://github.com/pandas-dev/pandas/blob/d91ffa6407c1baf6afe7d0a1b9655f44da77ac24/pandas/_libs/parsers.pyx#L1697 )? Is it more expensive to allocate/deallocate it for each function call ? It would be nice, if someone could explain this solution in more detail.

Finally, if pandas team decides to mark this issue `Won't fix`, maybe it can be specified in documentation, that `read_csv` doesn't interpret `Infinity` as ∞.

   




Hmm OK - thanks for the clarifications. Can you run the benchmarks in asv_bench/benchmarks/io/csv.py and see if this has an impact?
Sorry, I missed one file. Now it passes tests. I ran performance testing.  (Before fixing formatting)
```bash
asv continuous -E virtualenv --bench ^io master fix-#10065
```
In red color there is only this

```
     before           after         ratio
     [041b6b18]       [059808a0]
     <master>         <fix-#10065>
+      22.2±0.2ms         25.9±3ms     1.16  io.parsers.DoesStringLookLikeDatetime.time_check_datetimes('0.0')
+      35.0±0.3ms         39.2±2ms     1.12  io.parsers.DoesStringLookLikeDatetime.time_check_datetimes('10000')
```
Is this caused by my fix or some other commit? 
Full logs:
[results.zip](https://github.com/pandas-dev/pandas/files/3552720/results.zip)
[asv_full_output.txt](https://github.com/pandas-dev/pandas/files/3552730/asv_full_output.txt)

So, basically, my fix is the same as what  @gfyoung proposed in  https://github.com/pandas-dev/pandas/pull/13274 (closed in commit https://github.com/pandas-dev/pandas/commit/da5fc17fe9bee7527fb89b4540dc0c40555aaf16)  only for `Infinity`

 
So 15% increase on that one test - do we think that manifests itself typically from an end user perspective in any way? If not lgtm as well, just want to be careful of that
Thanks @githeap - great change