Repository: numpy/numpy

Issue:
Support floating point hex representation in loadtxt (Trac #1924)
_Original ticket http://projects.scipy.org/numpy/ticket/1924 on 2011-08-03 by trac user claumann, assigned to unknown._

Python supports a hexadecimal representation for floating numbers in float.hex() and float.fromhex():

> > > a = 3.14
> > > 
> > > a.hex()
> > > '0x1.91eb851eb851fp+1'
> > > 
> > > float.fromhex(a.hex())
> > > 3.14

Numpy's loadtxt does not support this hex text format but it seems like it would be a useful enhancement. For example, a simple implementation might come from changing the default converter returned by _getconv for columns of type float:

def _floatconv(x):
  try:
    return float(x)
  except ValueError:
    pass
  return float.fromhex(x)

And then change the return float in _getconv to return _floatconv. This converter obeys essentially the same semantics (and exceptions) as float() except that it falls back on fromhex before giving up.  

A slightly less permissive change would be to check that the string x begins with '0x' before calling fromhex and throwing a ValueError if not. The code above would accept 'bad' and convert it to 2989, which might be unnecessarily permissive.

Best, Chris



Discussion:

Resolving patch:
diff --git a/numpy/lib/npyio.py b/numpy/lib/npyio.py
--- a/numpy/lib/npyio.py
+++ b/numpy/lib/npyio.py
@@ -621,6 +621,13 @@ def _savez(file, args, kwds, compress):
 
 def _getconv(dtype):
     """ Find the correct dtype converter. Adapted from matplotlib """
+
+    def floatconv(x):
+        x.lower()
+        if b'0x' in x:
+            return float.fromhex(asstr(x))
+        return float(x)
+
     typ = dtype.type
     if issubclass(typ, np.bool_):
         return lambda x: bool(int(x))
@@ -631,7 +638,7 @@ def _getconv(dtype):
     if issubclass(typ, np.integer):
         return lambda x: int(float(x))
     elif issubclass(typ, np.floating):
-        return float
+        return floatconv
     elif issubclass(typ, np.complex):
         return complex
     elif issubclass(typ, np.bytes_):
@@ -706,6 +713,10 @@ def loadtxt(fname, dtype=float, comments='#', delimiter=None,
     `genfromtxt` function provides more sophisticated handling of, e.g.,
     lines with missing values.
 
+    .. versionadded:: 1.10.0
+    The strings produced by the Python float.hex method can be used as
+    input for floats.
+
     Examples
     --------
     >>> from StringIO import StringIO   # StringIO behaves like a file object


Test patch:
diff --git a/numpy/lib/tests/test_io.py b/numpy/lib/tests/test_io.py
--- a/numpy/lib/tests/test_io.py
+++ b/numpy/lib/tests/test_io.py
@@ -694,6 +694,19 @@ def test_int64_type(self):
         res = np.loadtxt(c, dtype=np.int64)
         assert_equal(res, tgt)
 
+    def test_from_float_hex(self):
+        # IEEE doubles and floats only, otherwise the float32
+        # conversion may fail.
+        tgt = np.logspace(-10, 10, 5).astype(np.float32)
+        tgt = np.hstack((tgt, -tgt)).astype(np.float)
+        inp = '\n'.join(map(float.hex, tgt))
+        c = TextIO()
+        c.write(inp)
+        for dt in [np.float, np.float32]:
+            c.seek(0)
+            res = np.loadtxt(c, dtype=dt)
+            assert_equal(res, tgt, err_msg="%s" % dt)
+
     def test_universal_newline(self):
         f, name = mkstemp()
         os.write(f, b'1 21\r3 42\r')