Repository: frictionlessdata/frictionless-py

Issue:
Provide an ability to work with float instead of decimal 
# Overview

I tried using `frictionless extract` on a small-ish mostly numeric dataset (~50,000 x 500), and it took a very long time (12.5 mins) to process the data.

Using the API directly was a bit faster, but still slow (2 mins).

I profiled the two different scenarios (attached). 

# To reproduce

Create some fake data:

```py
#!/bin/env python
import random
import pandas as pd
import numpy as np
from string import ascii_lowercase

# set random seeds
random.seed(1)
np.random.seed(1)

# create a fake data set with a single column of "ID's" and the rest numeric data
m = 50000
n = 500

# create random row ids
ids = [''.join(random.choice(ascii_lowercase) for i in range(10)) for j in range(m)]

# generate numeric portion of dataset and combine with row ids into a single dataframe
dat = pd.DataFrame(np.random.normal(0, 1, m * n).reshape([m, n]))
dat = pd.concat([pd.Series(ids), dat], axis=1)

# add colnames and store result
dat.columns = ["col_" + str(i) for i in range(n + 1)]

dat.to_csv("test_data.csv", index=False)
```

1. CLI call:

```
# normal call
frictionless extract test_data.csv

# to generate a profile
python -m cProfile -o profile_frictionless_cli.prof $HOME/.local/bin/frictionless extract test_data.csv
```

2. API call:

```py
# profile_frictionless.py
#!/bin/env python
from frictionless import extract
rows = extract('test_data.csv')

# to profile:
python -m cProfile -o profile_frictionless.prof profile_frictionless.py
```

# Observations

For the CLI calls, it seems like most of the time is spent in methods relating to pretty printing the tables (wcwidth.py and _pydecimal.py).

The actual number of calls is insanely large though -- for the example 50,000 x 500 dataset, there are 75,000,000 calls to to _pydecimal.py, which is 3x the amount of individual values in the dataset.

If you follow the smaller execution path (~180/770s) on the right-side, the result is similar. It goes:

```
main.py -> table.py -> table.py -> row.py -> field.py -> number.py -> _pydecimal.py
```

Calling the API directly basically looks like the smaller portion of the CLI profile, with a similar execution path to that shown above taking up most of the time.

I'm not sure how feasible it would be to try and optimize this / reduce the number of calls since, at a glance, it seems mostly to be from code outside of frictionless.

It could perhaps be possible to parallelize some of the code, although again, if it's just calls to some pretty print function that gets rendered directly to the screen, that also isn't likely to work / may be a pain to implement even if it could...

My cat would also like to add:

> qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq

Great point.

But more seriously, perhaps a better/much simpler solution would be to just have frictionless default to only printing the top `N` rows of data?

This is similar to how R/Pandas/Julia print matrices and dataframes to the screen.

You could similarly limit the number of columns displayed by default to make it more readable. If you could detect the terminal width, then the exact number could probably be optimized.

An option could then be included (e.g. `--num-rows`) to allow the user to override the default, with "0" indicating that all rows should be rendered.

Just a thought..

Thanks again for the help on discord! It is much appreciated.

Cheers,
Keith

# Snakeviz profiles

[profile_frictionless.prof.gz](https://github.com/frictionlessdata/frictionless-py/files/5343447/profile_frictionless.prof.gz)
[profile_frictionless_cli.prof.gz](https://github.com/frictionlessdata/frictionless-py/files/5343448/profile_frictionless_cli.prof.gz)

# System info

- Arch Linux 5.8.12 64-bit
- Python 3.8.5
- frictionless 3.19.2
- py-yaml 5.3.1

---

Please preserve this line to notify @roll (lead of this repository)


Discussion:

Patch:
diff --git a/frictionless/config.py b/frictionless/config.py
index 27589796..40900edc 100644
--- a/frictionless/config.py
+++ b/frictionless/config.py
@@ -58,6 +58,7 @@ DEFAULT_DATETIME_PATTERN = "%Y-%m-%dT%H:%M:%S%z"
 DEFAULT_DATE_PATTERN = "%Y-%m-%d"
 DEFAULT_TIME_PATTERN = "%H:%M:%S%z"
 DEFAULT_BARE_NUMBER = True
+DEFAULT_FLOAT_NUMBER = False
 DEFAULT_GROUP_CHAR = ""
 DEFAULT_DECIMAL_CHAR = "."
 DEFAULT_SERVER_PORT = 8000
diff --git a/frictionless/extract/table.py b/frictionless/extract/table.py
index 48210a1d..0a017344 100644
--- a/frictionless/extract/table.py
+++ b/frictionless/extract/table.py
@@ -26,6 +26,7 @@ def extract_table(
     infer_names=None,
     infer_volume=config.DEFAULT_INFER_VOLUME,
     infer_confidence=config.DEFAULT_INFER_CONFIDENCE,
+    infer_float_numbers=config.DEFAULT_FLOAT_NUMBER,
     infer_missing_values=config.DEFAULT_MISSING_VALUES,
     # Integrity
     onerror="ignore",
@@ -155,6 +156,8 @@ def extract_table(
         infer_names=infer_names,
         infer_volume=infer_volume,
         infer_confidence=infer_confidence,
+        infer_float_numbers=infer_float_numbers,
+        infer_missing_values=infer_missing_values,
         # Integrity
         onerror=onerror,
         lookup=lookup,
diff --git a/frictionless/field.py b/frictionless/field.py
index 750c566d..9e202b21 100644
--- a/frictionless/field.py
+++ b/frictionless/field.py
@@ -174,6 +174,14 @@ class Field(Metadata):
         """
         return self.get("bareNumber", config.DEFAULT_BARE_NUMBER)
 
+    @Metadata.property
+    def float_number(self):
+        """
+        Returns:
+            bool: whether it's a floating point number
+        """
+        return self.get("floatNumber", config.DEFAULT_FLOAT_NUMBER)
+
     @Metadata.property
     def decimal_char(self):
         """
diff --git a/frictionless/resource.py b/frictionless/resource.py
index 111bb252..97fc0c30 100644
--- a/frictionless/resource.py
+++ b/frictionless/resource.py
@@ -67,16 +67,29 @@ class Resource(Metadata):
         description=None,
         path=None,
         data=None,
+        # File
         scheme=None,
         format=None,
         hashing=None,
         encoding=None,
         compression=None,
         compression_path=None,
+        # Control/Dialect/Query
         control=None,
         dialect=None,
         query=None,
+        # Schema
         schema=None,
+        sync_schema=False,
+        patch_schema=None,
+        # Infer
+        infer_type=None,
+        infer_names=None,
+        infer_volume=config.DEFAULT_INFER_VOLUME,
+        infer_confidence=config.DEFAULT_INFER_CONFIDENCE,
+        infer_float_numbers=config.DEFAULT_FLOAT_NUMBER,
+        infer_missing_values=config.DEFAULT_MISSING_VALUES,
+        # Misc
         stats=None,
         profile=None,
         basepath=None,
@@ -109,6 +122,14 @@ class Resource(Metadata):
         self.setinitial("query", query)
         self.setinitial("schema", schema)
         self.setinitial("stats", stats)
+        self.__sync_schema = sync_schema
+        self.__patch_schema = patch_schema
+        self.__infer_type = infer_type
+        self.__infer_names = infer_names
+        self.__infer_volume = infer_volume
+        self.__infer_confidence = infer_confidence
+        self.__infer_float_numbers = infer_float_numbers
+        self.__infer_missing_values = infer_missing_values
         self.__basepath = basepath or helpers.detect_basepath(descriptor)
         self.__onerror = onerror
         self.__trusted = trusted
@@ -769,6 +790,14 @@ class Resource(Metadata):
         options.setdefault("dialect", self.dialect)
         options.setdefault("query", self.query)
         options.setdefault("schema", self.schema)
+        options.setdefault("sync_schema", self.__sync_schema)
+        options.setdefault("patch_schema", self.__patch_schema)
+        options.setdefault("infer_type", self.__infer_type)
+        options.setdefault("infer_names", self.__infer_names)
+        options.setdefault("infer_volume", self.__infer_volume)
+        options.setdefault("infer_confidence", self.__infer_confidence)
+        options.setdefault("infer_float_numbers", self.__infer_float_numbers)
+        options.setdefault("infer_missing_values", self.__infer_missing_values)
         options.setdefault("onerror", self.__onerror)
         if "lookup" not in options:
             options["lookup"] = self.read_lookup()
diff --git a/frictionless/schema.py b/frictionless/schema.py
index 76cabc79..a7b8cd93 100644
--- a/frictionless/schema.py
+++ b/frictionless/schema.py
@@ -183,6 +183,7 @@ class Schema(Metadata):
         type=None,
         names=None,
         confidence=config.DEFAULT_INFER_CONFIDENCE,
+        float_numbers=config.DEFAULT_FLOAT_NUMBER,
         missing_values=config.DEFAULT_MISSING_VALUES,
     ):
         """Infer schema
@@ -192,6 +193,7 @@ class Schema(Metadata):
             type? (str): enforce all the field to be the given type
             names (str[]): enforce field names
             confidence (float): infer confidence from 0 to 1
+            float_numbers (bool): infer numbers as `float` instead of `Decimal`
             missing_values (str[]): provide custom missing values
         """
 
@@ -232,6 +234,8 @@ class Schema(Metadata):
             candidates.append([])
             for type in INFER_TYPES:
                 field = Field(name=name, type=type, schema=self)
+                if type == "number" and float_numbers:
+                    field.float_number = True
                 candidates[index].append({"field": field, "score": 0})
             fields.append(Field(name=name, type="any", schema=self))
 
diff --git a/frictionless/table.py b/frictionless/table.py
index bef5ca3e..2bbc1424 100644
--- a/frictionless/table.py
+++ b/frictionless/table.py
@@ -101,6 +101,11 @@ class Table:
             For more information, please check "Describing  Data" guide.
             It defaults to 0.9
 
+        infer_float_numbers? (bool): Flag to indicate desired number type.
+            By default numbers will be `Decimal`; if `True` - `float`.
+            For more information, please check "Describing  Data" guide.
+            It defaults to `False`
+
         infer_missing_values? (str[]): String to be considered as missing values.
             For more information, please check "Describing  Data" guide.
             It defaults to `['']`
@@ -137,10 +142,12 @@ class Table:
         sync_schema=False,
         patch_schema=None,
         # Infer
+        # TODO: group as we did for Query?
         infer_type=None,
         infer_names=None,
         infer_volume=config.DEFAULT_INFER_VOLUME,
         infer_confidence=config.DEFAULT_INFER_CONFIDENCE,
+        infer_float_numbers=config.DEFAULT_FLOAT_NUMBER,
         infer_missing_values=config.DEFAULT_MISSING_VALUES,
         # Integrity
         onerror="ignore",
@@ -182,6 +189,7 @@ class Table:
         self.__infer_names = infer_names
         self.__infer_volume = infer_volume
         self.__infer_confidence = infer_confidence
+        self.__infer_float_numbers = infer_float_numbers
         self.__infer_missing_values = infer_missing_values
         self.__lookup = lookup
 
@@ -535,6 +543,7 @@ class Table:
                 type=self.__infer_type,
                 names=self.__infer_names or header,
                 confidence=self.__infer_confidence,
+                float_numbers=self.__infer_float_numbers,
                 missing_values=self.__infer_missing_values,
             )
 
diff --git a/frictionless/types/number.py b/frictionless/types/number.py
index 723fb95c..7afb7eab 100644
--- a/frictionless/types/number.py
+++ b/frictionless/types/number.py
@@ -23,21 +23,26 @@ class NumberType(Type):
     # Read
 
     def read_cell(self, cell):
+        Primary = Decimal
+        Secondary = float
+        if self.field.float_number:
+            Primary = float
+            Secondary = Decimal
         if isinstance(cell, str):
             if self.read_cell_processor:
                 cell = self.read_cell_processor(cell)
             try:
-                return Decimal(cell)
+                return Primary(cell)
             except Exception:
                 return None
-        elif isinstance(cell, Decimal):
+        elif isinstance(cell, Primary):
             return cell
         elif cell is True or cell is False:
             return None
         elif isinstance(cell, int):
             return cell
-        elif isinstance(cell, float):
-            return Decimal(str(cell))
+        elif isinstance(cell, Secondary):
+            return Primary(str(cell) if Primary is Decimal else cell)
         return None
 
     @Metadata.property(write=False)
diff --git a/frictionless/validate/table.py b/frictionless/validate/table.py
index b3976437..19751030 100644
--- a/frictionless/validate/table.py
+++ b/frictionless/validate/table.py
@@ -32,6 +32,7 @@ def validate_table(
     infer_names=None,
     infer_volume=config.DEFAULT_INFER_VOLUME,
     infer_confidence=config.DEFAULT_INFER_CONFIDENCE,
+    infer_float_numbers=config.DEFAULT_FLOAT_NUMBER,
     infer_missing_values=config.DEFAULT_MISSING_VALUES,
     # Integrity
     lookup=None,
@@ -183,6 +184,7 @@ def validate_table(
         infer_names=infer_names,
         infer_volume=infer_volume,
         infer_confidence=infer_confidence,
+        infer_float_numbers=infer_float_numbers,
         infer_missing_values=infer_missing_values,
         # Integrity
         lookup=lookup,


Test patch:
diff --git a/tests/test_resource.py b/tests/test_resource.py
index fcaf6e3c..beb2e652 100644
--- a/tests/test_resource.py
+++ b/tests/test_resource.py
@@ -461,82 +461,35 @@ def test_resource_schema_from_path_error_path_not_safe():
     assert error.note.count("schema.json")
 
 
-# Expand
-
-
-def test_resource_expand():
-    resource = Resource({"name": "name", "path": "data/table.csv"})
-    resource.expand()
-    assert resource == {
-        "name": "name",
-        "path": "data/table.csv",
-        "profile": "data-resource",
-    }
-
-
-def test_resource_expand_with_dialect():
-    dialect = {"delimiter": "custom"}
-    resource = Resource({"name": "name", "path": "data/table.csv", "dialect": dialect})
-    resource.expand()
-    assert resource == {
-        "name": "name",
-        "path": "data/table.csv",
-        "profile": "data-resource",
-        "dialect": {
-            "header": True,
-            "headerRows": [1],
-            "headerJoin": " ",
-            "headerCase": True,
-            "delimiter": "custom",
-            "lineTerminator": "\r\n",
-            "doubleQuote": True,
-            "quoteChar": '"',
-            "skipInitialSpace": False,
-        },
+def test_resource_sync_schema():
+    schema = {
+        "fields": [{"name": "name", "type": "string"}, {"name": "id", "type": "integer"}]
     }
+    resource = Resource(path="data/sync-schema.csv", schema=schema, sync_schema=True)
+    resource.infer()
+    assert resource.schema == schema
+    assert resource.read_header() == ["name", "id"]
+    assert resource.read_sample() == [["english", "1"], ["中国人", "2"]]
+    assert resource.read_rows() == [
+        {"id": 1, "name": "english"},
+        {"id": 2, "name": "中国人"},
+    ]
 
 
-def test_resource_expand_with_schema():
-    schema = {
+def test_table_schema_patch_schema():
+    patch_schema = {"fields": {"id": {"name": "new", "type": "string"}}}
+    resource = Resource(path="data/table.csv", patch_schema=patch_schema)
+    resource.infer()
+    assert resource.schema == {
         "fields": [
-            {"name": "id", "type": "integer"},
+            {"name": "new", "type": "string"},
             {"name": "name", "type": "string"},
-        ],
-    }
-    resource = Resource({"name": "name", "path": "data/table.csv", "schema": schema})
-    resource.expand()
-    assert resource == {
-        "name": "name",
-        "path": "data/table.csv",
-        "profile": "data-resource",
-        "schema": {
-            "fields": [
-                {
-                    "name": "id",
-                    "type": "integer",
-                    "format": "default",
-                    "bareNumber": True,
-                },
-                {"name": "name", "type": "string", "format": "default"},
-            ],
-            "missingValues": [""],
-        },
+        ]
     }
-
-
-# Write
-
-
-def test_resource_write(tmpdir):
-    path1 = "data/table.csv"
-    path2 = str(tmpdir.join("table.csv"))
-    source = Resource(path=path1)
-    source.write(path2)
-    target = Resource(path=path2, trusted=True)
-    assert target.read_header() == ["id", "name"]
-    assert target.read_rows() == [
-        {"id": 1, "name": "english"},
-        {"id": 2, "name": "中国人"},
+    assert resource.read_header() == ["id", "name"]
+    assert resource.read_rows() == [
+        {"new": "1", "name": "english"},
+        {"new": "2", "name": "中国人"},
     ]
 
 
@@ -613,6 +566,134 @@ def test_resource_infer_not_slugified_name_issue_531():
     assert resource.name == "table-with-data"
 
 
+def test_resource_infer_type():
+    resource = Resource(path="data/table.csv", infer_type="string")
+    resource.infer()
+    assert resource.schema == {
+        "fields": [
+            {"name": "id", "type": "string"},
+            {"name": "name", "type": "string"},
+        ]
+    }
+    assert resource.read_header() == ["id", "name"]
+    assert resource.read_rows() == [
+        {"id": "1", "name": "english"},
+        {"id": "2", "name": "中国人"},
+    ]
+
+
+def test_resource_infer_names():
+    resource = Resource(path="data/table.csv", infer_names=["new1", "new2"])
+    resource.infer()
+    assert resource.schema == {
+        "fields": [
+            {"name": "new1", "type": "integer"},
+            {"name": "new2", "type": "string"},
+        ]
+    }
+    assert resource.read_header() == ["id", "name"]
+    assert resource.read_rows() == [
+        {"new1": 1, "new2": "english"},
+        {"new1": 2, "new2": "中国人"},
+    ]
+
+
+def test_resource_infer_float_numbers():
+    data = [["number"], ["1.1"], ["2.2"], ["3.3"]]
+    resource = Resource(data=data, infer_float_numbers=True)
+    resource.infer()
+    assert resource.schema == {
+        "fields": [
+            {"name": "number", "type": "number", "floatNumber": True},
+        ]
+    }
+    assert resource.read_header() == ["number"]
+    assert resource.read_rows() == [
+        {"number": 1.1},
+        {"number": 2.2},
+        {"number": 3.3},
+    ]
+
+
+# Expand
+
+
+def test_resource_expand():
+    resource = Resource({"name": "name", "path": "data/table.csv"})
+    resource.expand()
+    assert resource == {
+        "name": "name",
+        "path": "data/table.csv",
+        "profile": "data-resource",
+    }
+
+
+def test_resource_expand_with_dialect():
+    dialect = {"delimiter": "custom"}
+    resource = Resource({"name": "name", "path": "data/table.csv", "dialect": dialect})
+    resource.expand()
+    assert resource == {
+        "name": "name",
+        "path": "data/table.csv",
+        "profile": "data-resource",
+        "dialect": {
+            "header": True,
+            "headerRows": [1],
+            "headerJoin": " ",
+            "headerCase": True,
+            "delimiter": "custom",
+            "lineTerminator": "\r\n",
+            "doubleQuote": True,
+            "quoteChar": '"',
+            "skipInitialSpace": False,
+        },
+    }
+
+
+def test_resource_expand_with_schema():
+    schema = {
+        "fields": [
+            {"name": "id", "type": "integer"},
+            {"name": "name", "type": "string"},
+        ],
+    }
+    resource = Resource({"name": "name", "path": "data/table.csv", "schema": schema})
+    resource.expand()
+    assert resource == {
+        "name": "name",
+        "path": "data/table.csv",
+        "profile": "data-resource",
+        "schema": {
+            "fields": [
+                {
+                    "name": "id",
+                    "type": "integer",
+                    "format": "default",
+                    "bareNumber": True,
+                },
+                {"name": "name", "type": "string", "format": "default"},
+            ],
+            "missingValues": [""],
+        },
+    }
+
+
+# Write
+
+
+def test_resource_write(tmpdir):
+    path1 = "data/table.csv"
+    path2 = str(tmpdir.join("table.csv"))
+    source = Resource(path=path1)
+    source.write(path2)
+    target = Resource(path=path2, trusted=True)
+    assert target.read_header() == ["id", "name"]
+    assert target.read_rows() == [
+        {"id": 1, "name": "english"},
+        {"id": 2, "name": "中国人"},
+    ]
+
+
 # Import/Export
 
 
diff --git a/tests/test_table.py b/tests/test_table.py
index d6e473ee..6b013de0 100644
--- a/tests/test_table.py
+++ b/tests/test_table.py
@@ -1064,7 +1064,10 @@ def test_table_schema_patch_schema_missing_values():
         ]
 
 
-def test_table_schema_infer_type():
+# Infer
+
+
+def test_table_infer_type():
     with Table("data/table.csv", infer_type="string") as table:
         assert table.header == ["id", "name"]
         assert table.schema == {
@@ -1079,7 +1082,7 @@ def test_table_schema_infer_type():
         ]
 
 
-def test_table_schema_infer_names():
+def test_table_infer_names():
     with Table("data/table.csv", infer_names=["new1", "new2"]) as table:
         assert table.header == ["id", "name"]
         assert table.schema == {
@@ -1094,6 +1097,9 @@ def test_table_schema_infer_names():
         ]
 
 
+# Integrity
+
+
 def test_table_schema_lookup_foreign_keys():
     source = [["name"], [1], [2], [3]]
     lookup = {"other": {("name",): {(1,), (2,), (3,)}}}
diff --git a/tests/types/test_number.py b/tests/types/test_number.py
index 5128b247..8580ca38 100644
--- a/tests/types/test_number.py
+++ b/tests/types/test_number.py
@@ -10,13 +10,16 @@ from frictionless import Field
     "format, source, target, options",
     [
         ("default", Decimal(1), Decimal(1), {}),
+        ("default", Decimal(1), 1, {"floatNumber": True}),
         ("default", 1, Decimal(1), {}),
         ("default", 1.0, Decimal(1), {}),
+        ("default", 1.0, 1.0, {"floatNumber": True}),
         ("default", 1 << 63, Decimal(1 << 63), {}),
         ("default", "1", Decimal(1), {}),
         ("default", "10.00", Decimal(10), {}),
         ("default", "10.50", Decimal(10.5), {}),
         ("default", 24.122667, Decimal("24.122667"), {}),
+        ("default", 24.122667, 24.122667, {"floatNumber": True}),
         ("default", "000835", Decimal("835"), {}),
         ("default", "100%", Decimal(100), {"bareNumber": False}),
         ("default", "1000‰", Decimal(1000), {"bareNumber": False}),