Repository: numpy/numpy

Issue:
BUG:  not assigning full precision values to longdouble scalars
If you try to set a longdouble scalar with a string, it appears to go through Python's normal string-parsing and therefore gets converted to a C double rather than a long double:

In [19]: numpy.longdouble("1.12345678901234567890")
Out[19]: 1.1234567890123456912

Note the loss of those last couple digits.

In [20]: numpy.float("1.12345678901234567890")
Out[20]: 1.1234567890123457

In [21]: numpy.longdouble("1.12345678901234567890") - numpy.float("1.12345678901234567890")
Out[21]: 0.0

And so internally they are identical.

One solution to this is a special case for longdoubles and to use strtold() to do the conversion.

Cheers,

Scott

BUG:  not assigning full precision values to longdouble scalars
If you try to set a longdouble scalar with a string, it appears to go through Python's normal string-parsing and therefore gets converted to a C double rather than a long double:

In [19]: numpy.longdouble("1.12345678901234567890")
Out[19]: 1.1234567890123456912

Note the loss of those last couple digits.

In [20]: numpy.float("1.12345678901234567890")
Out[20]: 1.1234567890123457

In [21]: numpy.longdouble("1.12345678901234567890") - numpy.float("1.12345678901234567890")
Out[21]: 0.0

And so internally they are identical.

One solution to this is a special case for longdoubles and to use strtold() to do the conversion.

Cheers,

Scott

fromstring/fromfile behave badly with malformed data (Trac #883)
_Original ticket http://projects.scipy.org/numpy/ticket/883 on 2008-08-09 by @pv, assigned to unknown._

See the discussion at:

```
http://permalink.gmane.org/gmane.comp.python.numeric.general/24115
```

In summary: 
1. Empty string is understood as zero (except at the end of a string)
   
   > > > np.fromstring('xxxxx', sep='x')
   > > > array([ 0.,  0.,  0.,  0.,  0.])
   
   Missing values are treated as zeros. This is probably accidental, it's more likely that the memory allocated just happens to be filled with zeros...
2. If count is given and not enough entries were read, the rest is padded with uninitialized data
   
   > > > np.fromstring("23", dtype=np.float, sep="x", count=3)
   > > > array([  2.30000000e+001,   8.19397657e-269,   1.94625366e-312])
3. Fromfile behaves differently from fromstring (Assume "foo.dat" contains string "23xxx")
   
   > > > np.fromfile("foo.dat", dtype=np.float, sep="x")
   > > > array([  2.30000000e+001,   8.36783574e-269,   2.12885402e-313])
   > > > np.fromfile("foo.dat", dtype=np.float, count=5, sep="x")
   > > > 5 items requested but only 3 read
   > > > array([  2.30000000e+001,   6.04468244e-269,  -1.54891786e-039])
   
   Again, the last value is ignored, but the former ones are not treated as zero; instead, the array is filled with uninitialized data for the missing entries. Note also that neither warning or error is raised; instead, a message is printed to stdout.

Question: what is the correct behavior:

A) Raise ValueError on malformed data.

B) Raise ValueError on malformed data if count is given, otherwise truncate to valid part.

C) Truncate to valid part of data.



Discussion:
Yeah, this is a long standing problem. There may even be other issues opened for it.

this one is in principle easy to fixi n LONGDOUBLE_setitem, if we could assume strtold is available
unfortunately this is only guaranteed by C99

sscanf could be used, in principle anyway: K&R describes %Lf as a valid scanf speicifier, so that should work in every reasonable C environment.

Yeah, this is a long standing problem. There may even be other issues opened for it.

this one is in principle easy to fixi n LONGDOUBLE_setitem, if we could assume strtold is available
unfortunately this is only guaranteed by C99

sscanf could be used, in principle anyway: K&R describes %Lf as a valid scanf speicifier, so that should work in every reasonable C environment.

Milestone changed to `1.4.0` by @cournape on 2009-03-11

_@charris wrote on 2009-05-24_

Note that the behavior i 3) has changed as of the current release (1.3.0). With the same file containing "23xxx"

```
In [1]: fromfile("tmp", sep="x")
Out[1]: array([ 23.,   0.,   0.,   0.])

In [2]: fromfile("tmp", count=5, sep="x")
5 items requested but only 4 read
Out[2]: array([ 23.,   0.,   0.,   0.])
```

Fromfile and fromstring behave differently, however. At the moment I'm guessing it comes from differences in fromstr_skip_separator and fromfile_skip_separator but I haven't looked closely at these functions yet.

Milestone changed to `Unscheduled` by @mwiebe on 2011-03-23

Still open in 1.9-devel.


Resolving patch:
diff --git a/numpy/core/setup_common.py b/numpy/core/setup_common.py
--- a/numpy/core/setup_common.py
+++ b/numpy/core/setup_common.py
@@ -104,7 +104,7 @@ def check_api_version(apiversion, codegen_dir):
 OPTIONAL_STDFUNCS = ["expm1", "log1p", "acosh", "asinh", "atanh",
         "rint", "trunc", "exp2", "log2", "hypot", "atan2", "pow",
         "copysign", "nextafter", "ftello", "fseeko",
-        "strtoll", "strtoull", "cbrt"]
+        "strtoll", "strtoull", "cbrt", "strtold_l",]
 
 
 OPTIONAL_HEADERS = [
diff --git a/numpy/lib/_iotools.py b/numpy/lib/_iotools.py
--- a/numpy/lib/_iotools.py
+++ b/numpy/lib/_iotools.py
@@ -527,6 +527,7 @@ class StringConverter(object):
 
     _mapper.extend([(nx.floating, float, nx.nan),
                     (complex, _bytes_to_complex, nx.nan + 0j),
+                    (nx.longdouble, nx.longdouble, nx.nan),
                     (nx.string_, bytes, asbytes('???'))])
 
     (_defaulttype, _defaultfunc, _defaultfill) = zip(*_mapper)
@@ -643,6 +644,18 @@ def __init__(self, dtype_or_func=None, default=None, missing_values=None,
                     else:
                         self.default = default
                     break
+            # if a converter for the specific dtype is available use that
+            last_func = func
+            for (i, (deftype, func, default_def)) in enumerate(self._mapper):
+                if dtype.type == deftype:
+                    _status = i
+                    last_func = func
+                    if default is None:
+                        self.default = default_def
+                    else:
+                        self.default = default
+                    break
+            func = last_func
             if _status == -1:
                 # We never found a match in the _mapper...
                 _status = 0
diff --git a/numpy/lib/npyio.py b/numpy/lib/npyio.py
--- a/numpy/lib/npyio.py
+++ b/numpy/lib/npyio.py
@@ -667,6 +667,8 @@ def floatconv(x):
         return np.int64
     if issubclass(typ, np.integer):
         return lambda x: int(float(x))
+    elif issubclass(typ, np.longdouble):
+        return np.longdouble
     elif issubclass(typ, np.floating):
         return floatconv
     elif issubclass(typ, np.complex):


Test patch:
diff --git a/numpy/core/tests/test_longdouble.py b/numpy/core/tests/test_longdouble.py
new file mode 100644
--- /dev/null
+++ b/numpy/core/tests/test_longdouble.py
@@ -0,0 +1,215 @@
+from __future__ import division, absolute_import, print_function
+
+import locale
+from tempfile import NamedTemporaryFile
+
+import numpy as np
+from numpy.testing import (
+    run_module_suite, assert_, assert_equal, dec, assert_raises,
+    assert_array_equal, TestCase
+)
+from numpy.compat import sixu
+from test_print import in_foreign_locale
+
+longdouble_longer_than_double = (np.finfo(np.longdouble).eps
+                                 < np.finfo(np.double).eps)
+
+
+_o = 1 + np.finfo(np.longdouble).eps
+string_to_longdouble_inaccurate = (_o != np.longdouble(repr(_o)))
+del _o
+
+def test_scalar_extraction():
+    """Confirm that extracting a value doesn't convert to python float"""
+    o = 1 + np.finfo(np.longdouble).eps
+    a = np.array([o, o, o])
+    assert_equal(a[1], o)
+
+
+# Conversions string -> long double
+
+
+def test_repr_roundtrip():
+    o = 1 + np.finfo(np.longdouble).eps
+    assert_equal(np.longdouble(repr(o)), o,
+                 "repr was %s" % repr(o))
+
+
+def test_unicode():
+    np.longdouble(sixu("1.2"))
+
+
+def test_string():
+    np.longdouble("1.2")
+
+
+def test_bytes():
+    np.longdouble(b"1.2")
+
+
+@in_foreign_locale
+def test_fromstring_foreign():
+    f = 1.234
+    a = np.fromstring(repr(f), dtype=float, sep=" ")
+    assert_equal(a[0], f)
+
+
+@dec.knownfailureif(string_to_longdouble_inaccurate, "Need strtold_l")
+def test_repr_roundtrip_bytes():
+    o = 1 + np.finfo(np.longdouble).eps
+    assert_equal(np.longdouble(repr(o).encode("ascii")), o)
+
+
+@in_foreign_locale
+def test_repr_roundtrip_foreign():
+    o = 1.5
+    assert_equal(o, np.longdouble(repr(o)))
+
+
+def test_bogus_string():
+    assert_raises(ValueError, np.longdouble, "spam")
+    assert_raises(ValueError, np.longdouble, "1.0 flub")
+
+
+@dec.knownfailureif(string_to_longdouble_inaccurate, "Need strtold_l")
+def test_fromstring():
+    o = 1 + np.finfo(np.longdouble).eps
+    s = (" " + repr(o))*5
+    a = np.array([o]*5)
+    assert_equal(np.fromstring(s, sep=" ", dtype=np.longdouble), a,
+                 err_msg="reading '%s'" % s)
+
+
+def test_fromstring_bogus():
+    assert_equal(np.fromstring("1. 2. 3. flop 4.", dtype=float, sep=" "),
+                 np.array([1., 2., 3.]))
+
+
+def test_fromstring_empty():
+    assert_equal(np.fromstring("xxxxx", sep="x"),
+                 np.array([]))
+
+
+def test_fromstring_missing():
+    assert_equal(np.fromstring("1xx3x4x5x6", sep="x"),
+                 np.array([1]))
+
+
+class FileBased(TestCase):
+    def setUp(self):
+        self.o = 1 + np.finfo(np.longdouble).eps
+        self.f = NamedTemporaryFile(mode="wt")
+
+    def tearDown(self):
+        self.f.close()
+        del self.f
+
+    def test_fromfile_bogus(self):
+        self.f.write("1. 2. 3. flop 4.\n")
+        self.f.flush()
+        F = open(self.f.name, "rt")
+        try:
+            assert_equal(np.fromfile(F, dtype=float, sep=" "),
+                         np.array([1., 2., 3.]))
+        finally:
+            F.close()
+
+    @dec.knownfailureif(string_to_longdouble_inaccurate, "Need strtold_l")
+    def test_fromfile(self):
+        for i in range(5):
+            self.f.write(repr(self.o) + "\n")
+        self.f.flush()
+        a = np.array([self.o]*5)
+        F = open(self.f.name, "rt")
+        b = np.fromfile(F,
+                        dtype=np.longdouble,
+                        sep="\n")
+        F.close()
+        F = open(self.f.name, "rt")
+        s = F.read()
+        F.close()
+        assert_equal(b, a, err_msg="decoded %s as %s" % (repr(s), repr(b)))
+
+    @dec.knownfailureif(string_to_longdouble_inaccurate, "Need strtold_l")
+    def test_genfromtxt(self):
+        for i in range(5):
+            self.f.write(repr(self.o) + "\n")
+        self.f.flush()
+        a = np.array([self.o]*5)
+        assert_equal(np.genfromtxt(self.f.name, dtype=np.longdouble), a)
+
+    @dec.knownfailureif(string_to_longdouble_inaccurate, "Need strtold_l")
+    def test_loadtxt(self):
+        for i in range(5):
+            self.f.write(repr(self.o) + "\n")
+        self.f.flush()
+        a = np.array([self.o]*5)
+        assert_equal(np.loadtxt(self.f.name, dtype=np.longdouble), a)
+
+    @dec.knownfailureif(string_to_longdouble_inaccurate, "Need strtold_l")
+    def test_tofile_roundtrip(self):
+        a = np.array([self.o]*3)
+        a.tofile(self.f.name, sep=" ")
+        F = open(self.f.name, "rt")
+        try:
+            assert_equal(np.fromfile(F, dtype=np.longdouble, sep=" "),
+                         a)
+        finally:
+            F.close()
+
+
+@in_foreign_locale
+def test_fromstring_foreign():
+    s = "1.234"
+    a = np.fromstring(s, dtype=np.longdouble, sep=" ")
+    assert_equal(a[0], np.longdouble(s))
+
+
+@in_foreign_locale
+def test_fromstring_foreign_sep():
+    a = np.array([1, 2, 3, 4])
+    b = np.fromstring("1,2,3,4,", dtype=np.longdouble, sep=",")
+    assert_array_equal(a, b)
+
+
+@in_foreign_locale
+def test_fromstring_foreign_value():
+    b = np.fromstring("1,234", dtype=np.longdouble, sep=" ")
+    assert_array_equal(b[0], 1)
+
+
+# Conversions long double -> string
+
+
+def test_repr_exact():
+    o = 1 + np.finfo(np.longdouble).eps
+    assert_(repr(o) != '1')
+
+
+@dec.knownfailureif(longdouble_longer_than_double, "BUG #2376")
+@dec.knownfailureif(string_to_longdouble_inaccurate, "Need strtold_l")
+def test_format():
+    o = 1 + np.finfo(np.longdouble).eps
+    assert_("{0:.40g}".format(o) != '1')
+
+
+@dec.knownfailureif(longdouble_longer_than_double, "BUG #2376")
+@dec.knownfailureif(string_to_longdouble_inaccurate, "Need strtold_l")
+def test_percent():
+    o = 1 + np.finfo(np.longdouble).eps
+    assert_("%.40g" % o != '1')
+
+
+@dec.knownfailureif(longdouble_longer_than_double, "array repr problem")
+@dec.knownfailureif(string_to_longdouble_inaccurate, "Need strtold_l")
+def test_array_repr():
+    o = 1 + np.finfo(np.longdouble).eps
+    a = np.array([o])
+    b = np.array([1], dtype=np.longdouble)
+    if not np.all(a != b):
+        raise ValueError("precision loss creating arrays")
+    assert_(repr(a) != repr(b))
+
+
+if __name__ == "__main__":
+    run_module_suite()
diff --git a/numpy/core/tests/test_multiarray.py b/numpy/core/tests/test_multiarray.py
--- a/numpy/core/tests/test_multiarray.py
+++ b/numpy/core/tests/test_multiarray.py
@@ -3411,7 +3411,9 @@ def test_tofile_sep(self):
         f = open(self.filename, 'r')
         s = f.read()
         f.close()
-        assert_equal(s, '1.51,2.0,3.51,4.0')
+        #assert_equal(s, '1.51,2.0,3.51,4.0')
+        y = np.array([float(p) for p in s.split(',')])
+        assert_array_equal(x,y)
 
     def test_tofile_format(self):
         x = np.array([1.51, 2, 3.51, 4], dtype=float)