Repository: frictionlessdata/frictionless-py

Issue:
Add validation summary to the CLI report
# Overview

Currently, we have:

```
# -------
# invalid: data/invalid.csv
# -------

===  =====  ===============  ====================================================================================
row  field  code             message
===  =====  ===============  ====================================================================================
         3  blank-label      Label in the header in field at position "3" is blank
         4  duplicate-label  Label "name" in the header at position "4" is duplicated to a label: at position "2"
  2      3  missing-cell     Row at position "2" has a missing cell in field "field3" at position "3"
  2      4  missing-cell     Row at position "2" has a missing cell in field "name2" at position "4"
  3      3  missing-cell     Row at position "3" has a missing cell in field "field3" at position "3"
  3      4  missing-cell     Row at position "3" has a missing cell in field "name2" at position "4"
  4         blank-row        Row at position "4" is completely blank
  5      5  extra-cell       Row at position "5" has an extra value in field at position "5"
===  =====  ===============  ====================================================================================
```

We'd like to extend the report to something like:

```
# -------
# invalid: data/invalid.csv
# -------

# Summary

[table with stats]

# Errors

===  =====  ===============  ====================================================================================
row  field  code             message
===  =====  ===============  ====================================================================================
         3  blank-label      Label in the header in field at position "3" is blank
         4  duplicate-label  Label "name" in the header at position "4" is duplicated to a label: at position "2"
  2      3  missing-cell     Row at position "2" has a missing cell in field "field3" at position "3"
  2      4  missing-cell     Row at position "2" has a missing cell in field "name2" at position "4"
  3      3  missing-cell     Row at position "3" has a missing cell in field "field3" at position "3"
  3      4  missing-cell     Row at position "3" has a missing cell in field "name2" at position "4"
  4         blank-row        Row at position "4" is completely blank
  5      5  extra-cell       Row at position "5" has an extra value in field at position "5"
===  =====  ===============  ====================================================================================
```

# Plan

- [ ] brainstorm the summary table contents
- [ ] agree with @roll @shashigharti 
- [ ] implement


Discussion:
roll: Hi @shashigharti, can you please add a screenshot/output?
shashigharti: Hi @roll 
Here are the screenshots:
Summary
![summary](https://user-images.githubusercontent.com/5582809/170660189-c2b0c30e-09e8-43ff-8b43-fda71558fe1a.png)

Summary with partial info
![partial](https://user-images.githubusercontent.com/5582809/170660202-d5afdb3e-b0e4-4f05-976a-a40785cea6c0.png)


shashigharti: Screenshot for valid resource:
![valid](https://user-images.githubusercontent.com/5582809/170660645-8dbc12c4-de89-46bc-ae9b-b8b547329be0.png)



Patch:
diff --git a/data/fixtures/cli/long-error-messages-976.txt b/data/fixtures/cli/long-error-messages-976.txt
index 5617a77e..2ba67bba 100644
--- a/data/fixtures/cli/long-error-messages-976.txt
+++ b/data/fixtures/cli/long-error-messages-976.txt
@@ -2,6 +2,19 @@
 # invalid: test-tabulator 
 # -------
 
+
+# Summary 
+
+Description                      Size/Name/Count
+-------------------------------  -----------------
+File name (Not Found)            test-tabulator
+File size                        N/A
+Total Time Taken (sec)           
+Total Errors                     1
+Resource Error (resource-error)  1
+
+# Errors 
+
 row    field    code       message
 -----  -------  ---------  -------------------------------------------------
                 resource-  The data resource has an error: "{'format':
diff --git a/data/fixtures/cli/zipped-resources-979.txt b/data/fixtures/cli/zipped-resources-979.txt
index 9cd498c4..de9b7ef3 100644
--- a/data/fixtures/cli/zipped-resources-979.txt
+++ b/data/fixtures/cli/zipped-resources-979.txt
@@ -1,10 +1,31 @@
 # -----
 # valid: ogd10_energieforschungstatistik_ch.csv 
 # -----
+
+# Summary 
+
+Description             Size/Name/Count
+----------------------  --------------------------------------
+File name               ogd10_energieforschungstatistik_ch.csv
+File size (KB)          88541
+Total Time Taken (sec)  
 # -------
 # invalid: ogd10_catalogs.zip => capital-invalid.csv 
 # -------
 
+
+# Summary 
+
+Description                  Size/Name/Count
+---------------------------  -----------------------------------------
+File name (Not Found)        ogd10_catalogs.zip => capital-invalid.csv
+File size                    N/A
+Total Time Taken (sec)       
+Total Errors                 1
+Schema Error (schema-error)  1
+
+# Errors 
+
 row    field    code     message
 -----  -------  -------  -------------------------------------------------
                 schema-  Schema is not valid: Schemas with duplicate field
@@ -12,4 +33,12 @@ row    field    code     message
 
 # -----
 # valid: ogd10_catalogs.zip => finanzquellen.csv 
-# -----
\ No newline at end of file
+# -----
+
+# Summary 
+
+Description             Size/Name/Count
+----------------------  ---------------------------------------
+File name (Not Found)   ogd10_catalogs.zip => finanzquellen.csv
+File size               N/A
+Total Time Taken (sec)  
diff --git a/data/issue-1105-timezone.csv b/data/issue-1105-timezone.csv
new file mode 100644
index 00000000..cc6f198b
--- /dev/null
+++ b/data/issue-1105-timezone.csv
@@ -0,0 +1,3 @@
+id;timestamp
+0;2020-01-01T15:00:00Z
+1;2020-01-01T15:00:00+03:00
diff --git a/data/issue-1105.package.json b/data/issue-1105.package.json
new file mode 100644
index 00000000..6f401ba8
--- /dev/null
+++ b/data/issue-1105.package.json
@@ -0,0 +1,24 @@
+{
+    "name": "to_pandas_bug",
+    "resources": [
+        {
+            "profile": "tabular-data-resource",
+            "name": "issue1105",
+            "path": "issue-1105-timezone.csv",
+            "scheme": "file",
+            "format": "csv",
+            "encoding": "utf-8",
+            "schema": {
+                "fields": [
+                    {"name": "id", "type": "integer"},
+                    {
+                        "name": "timestamp",
+                        "type": "datetime"
+                    }
+                ],
+                "primaryKey": ["timestamp"]
+            },
+            "dialect": {"delimiter": ";"}
+        }
+    ]
+}
diff --git a/frictionless/helpers.py b/frictionless/helpers.py
index aba54a4f..8686b11e 100644
--- a/frictionless/helpers.py
+++ b/frictionless/helpers.py
@@ -6,6 +6,7 @@ import json
 import glob
 import jinja2
 import marko
+import math
 import atexit
 import shutil
 import zipfile
@@ -631,3 +632,12 @@ def dicts_to_markdown_table(dicts: List[dict], **kwargs) -> str:
         error = errors.GeneralError(note="Please install `pandas` package")
         raise module.FrictionlessException(error)
     return df.where(df.notnull(), None).to_markdown(index=False)
+
+
+def format_bytes(size: int) -> str:
+    """Format bytes to larger units"""
+    units = ["bytes", "KB", "MB", "GB", "TB"]
+    index = math.floor(math.log2(size) / 10)
+    if index > len(units):
+        index = len(units) - 1
+    return units[index]
diff --git a/frictionless/plugins/pandas/parser.py b/frictionless/plugins/pandas/parser.py
index a4ed9213..23fcf4a8 100644
--- a/frictionless/plugins/pandas/parser.py
+++ b/frictionless/plugins/pandas/parser.py
@@ -141,6 +141,8 @@ class PandasParser(Parser):
                 data_rows.append(tuple(data_values))
 
         # Create index
+        pd = helpers.import_from_plugin("pandas", plugin="pandas")
+
         index = None
         if source.schema.primary_key:
             if len(source.schema.primary_key) == 1:
@@ -149,7 +151,9 @@ class PandasParser(Parser):
                 index_dtype = self.__write_convert_type(index_field.type)
                 if index_field.type in ["datetime", "date"]:
                     index_class = pd.DatetimeIndex
+                    index_rows = pd.to_datetime(index_rows, utc=True)
                 index = index_class(index_rows, name=index_field.name, dtype=index_dtype)
+
             elif len(source.schema.primary_key) > 1:
                 index = pd.MultiIndex.from_tuples(
                     index_rows, names=source.schema.primary_key
@@ -167,12 +171,13 @@ class PandasParser(Parser):
 
     def __write_convert_type(self, type=None):
         np = helpers.import_from_plugin("numpy", plugin="pandas")
+        pd = helpers.import_from_plugin("pandas", plugin="pandas")
 
         # Mapping
         mapping = {
             "array": np.dtype(list),
             "boolean": np.dtype(bool),
-            "datetime": np.dtype("datetime64[ns]"),
+            "datetime": pd.DatetimeTZDtype(tz="UTC"),
             "integer": np.dtype(int),
             "number": np.dtype(float),
             "object": np.dtype(dict),
diff --git a/frictionless/program/validate.py b/frictionless/program/validate.py
index 06004fb9..67aeadfd 100644
--- a/frictionless/program/validate.py
+++ b/frictionless/program/validate.py
@@ -1,5 +1,5 @@
 import sys
-import petl
+import os
 import typer
 import textwrap
 from typing import List
@@ -226,11 +226,7 @@ def program_validate(
         for error in report.errors:
             content.append([error.code, error.message])
         typer.secho(
-            str(
-                petl.util.vis.lookall(
-                    [["code", "message"]] + content, vrepr=str, style="simple"
-                )
-            )
+            str(tabulate(content, headers=["code", "message"], tablefmt="simple"))
         )
 
     # Return tables
@@ -250,6 +246,7 @@ def program_validate(
         typer.secho(f"# {'-'*len(prefix)}", bold=True)
         typer.secho(f"# {prefix}: {source} {suffix}", bold=True)
         typer.secho(f"# {'-'*len(prefix)}", bold=True)
+        error_list = {}
         if task.errors:
             prev_invalid = True
             typer.secho("")
@@ -263,7 +260,42 @@ def program_validate(
                         error.message,
                     ]
                 )
+                # error list for summary
+                error_title = f"{error.name} ({error.code})"
+                if error_title not in error_list:
+                    error_list[error_title] = 0
+                error_list[error_title] += 1
+                if task.partial:
+                    last_row_checked = error.get("rowPosition", "")
             content = _wrap_text_to_colwidths(content)
+        # summary
+        rows_checked = last_row_checked if task.partial else None
+        summary_content = _validation_summary(
+            source,
+            basepath=task.resource.basepath,
+            time_taken=task.time,
+            rows_checked=rows_checked,
+            error_list=error_list,
+        )
+        typer.echo("\n# Summary \n")
+        if task.partial:
+            typer.echo(
+                "The document was partially validated because of one of the limits"
+            )
+            typer.echo("* limit errors")
+            typer.echo("* memomry Limit \n")
+        typer.secho(
+            str(
+                tabulate(
+                    summary_content,
+                    headers=["Description", "Size/Name/Count"],
+                    tablefmt="simple",
+                )
+            )
+        )
+        # errors
+        if task.errors:
+            typer.echo("\n# Errors \n")
             typer.secho(
                 str(
                     tabulate(
@@ -301,3 +333,32 @@ def _wrap_text_to_colwidths(
             new_row.append("\n".join(wrapped))
         result.append(new_row)
     return result
+
+
+def _validation_summary(
+    source: str,
+    time_taken: str,
+    basepath: str = None,
+    rows_checked: int = None,
+    error_list: List = None,
+) -> List:
+    """Generate summary for validation task"""
+    file_path = os.path.join(basepath, source) if basepath else source
+    file_size = "N/A"
+    unit = None
+    if os.path.exists(file_path):
+        file_size = os.path.getsize(file_path)
+        unit = helpers.format_bytes(file_size)
+    content = [
+        [f"File name { '' if unit else '(Not Found)' }", source],
+        [f"File size { f'({unit})' if unit else '' }", file_size],
+        ["Total Time Taken (sec)", time_taken],
+    ]
+    if rows_checked:
+        content.append(["Rows Checked(Partial)**", rows_checked])
+    if error_list:
+        content.append(["Total Errors", sum(error_list.values())])
+    for code, count in error_list.items():
+        content.append([code, count])
+
+    return content


Test patch:
diff --git a/tests/detector/test_general.py b/tests/detector/test_general.py
index 6bdb8f33..959acad4 100644
--- a/tests/detector/test_general.py
+++ b/tests/detector/test_general.py
@@ -1,4 +1,5 @@
 from frictionless import Detector, Resource
+import pytest
 
 # General
 
@@ -79,6 +80,55 @@ def test_schema_from_sparse_sample():
     }
 
 
+@pytest.mark.parametrize("confidence", [0.6, 0.7, 0.8])
+def test_schema_from_synthetic_sparse_sample(confidence):
+
+    # For each type (integer, number, string) there are example of
+    # the type ("is") and examples of other type ("not")
+    type_sample = {
+        "integer": {"is": 1, "not": "string"},
+        "number": {"is": 3.14, "not": "string"},
+        "string": {"is": "string", "not": 1},
+    }
+
+    # Columns with type and confidence
+    columns = [
+        {"type": "integer", "conf": 0.7},
+        {"type": "number", "conf": 1},
+        {"type": "string", "conf": 1},
+    ]
+
+    def generate_rows(num_rows=100, columns=[]):
+        rows = []
+        num_per_type = [num_rows * c["conf"] for c in columns]
+
+        for i in range(num_rows):
+            row = []
+            for ci, col in enumerate(columns):
+                if i < num_per_type[ci]:
+                    row.append(type_sample[col["type"]]["is"])
+                else:
+                    row.append(type_sample[col["type"]]["not"])
+
+            rows.append(row)
+
+        return rows
+
+    sample = generate_rows(columns=columns)
+    detector = Detector(field_confidence=confidence)
+    labels = [f"field{i}" for i in range(1, 4)]
+    schema = detector.detect_schema(sample, labels=labels)
+    assert schema == {
+        "fields": [
+            {
+                "name": f"field{i + 1}",
+                "type": columns[i]["type"] if columns[i]["conf"] >= confidence else "any",
+            }
+            for i in range(len(columns))
+        ],
+    }
+
+
 def test_schema_infer_no_names():
     sample = [[1], [2], [3]]
     detector = Detector()
diff --git a/tests/plugins/pandas/test_parser.py b/tests/plugins/pandas/test_parser.py
index 84a4808a..7c2342dc 100644
--- a/tests/plugins/pandas/test_parser.py
+++ b/tests/plugins/pandas/test_parser.py
@@ -42,6 +42,17 @@ def test_pandas_parser_write_bug_1100():
     ]
 
 
+def test_pandas_parser_write_bug_1105():
+    datapackage = Package("data/issue-1105.package.json")
+    target = datapackage.resources[0].to_pandas()
+    assert target.to_dict() == {
+        "id": {
+            pd.Timestamp("2020-01-01 12:00:00+0000", tz="UTC"): 1,
+            pd.Timestamp("2020-01-01 15:00:00+0000", tz="UTC"): 0,
+        }
+    }
+
+
 def test_pandas_parser_write_types():
     source = Package("data/storage/types.json").get_resource("types")
     target = source.write(format="pandas")
diff --git a/tests/program/test_validate.py b/tests/program/test_validate.py
index 90804564..b755f781 100644
--- a/tests/program/test_validate.py
+++ b/tests/program/test_validate.py
@@ -1,5 +1,6 @@
 import json
 import yaml
+import re
 from typer.testing import CliRunner
 from frictionless import Metadata, Detector, program, validate
 
@@ -223,7 +224,9 @@ def test_program_validate_zipped_resources_979():
     assert result.stdout.count("valid: ogd10_catalogs.zip => finanzquellen.csv")
     assert result.stdout.count("invalid: ogd10_catalogs.zip => capital-invalid.csv")
     assert result.stdout.count("Schema is not valid")
-    assert result.stdout.strip() == expected.strip()
+    # remove timetaken floating point number which varies
+    output = re.sub(r"(\d+)\.(.*)\d", "", result.stdout)
+    assert output.strip() == expected.strip()
 
 
 def test_program_validate_long_error_messages_976():
@@ -231,8 +234,27 @@ def test_program_validate_long_error_messages_976():
     output_file_path = "data/fixtures/cli/long-error-messages-976.txt"
     with open(output_file_path, encoding="utf-8") as file:
         expected = file.read()
+    output = re.sub(r"(\d+)\.(.*)\d", "", result.stdout)
     assert result.exit_code == 1
-    assert result.stdout.strip() == expected.strip()
+    assert output.strip() == expected.strip()
+
+
+def test_program_validate_partial_validation_info_933():
+    result = runner.invoke(program, "validate data/countries.csv --limit-errors 2")
+    assert result.exit_code == 1
+    assert result.stdout.count(
+        "The document was partially validated because of one of the limits"
+    )
+    assert result.stdout.count("Rows Checked(Partial)")
+
+
+def test_program_validate_summary_1094():
+    result = runner.invoke(program, "validate data/datapackage.json --type resource")
+    assert result.exit_code == 1
+    assert result.stdout.count("Summary")
+    assert result.stdout.count("File name")
+    assert result.stdout.count("File size")
+    assert result.stdout.count("Total Time Taken (sec)")
 
 
 # Helpers