Repository: Arelle/Arelle

Issue:
Update insignificantDigits to handle large numbers
### What should we change and why?

Hi, Hope all is well! 

We ran into a numeric fact value that was too large for ValidateXbrlCalcs.py (388163667900000000000000000). This caused an exception (please see below) in Arelle. This number was proforma and not actual. I recommend either using a data type that can handle larger numbers or add a test that will alleviate exceptions and provide feedback to the user to easily identify what needs to be investigated and/or fixed. Further details below. Thank you :)

Regarding the file below:

Arelle/Arelle/ValidateXbrlCalcs.py

Line 741, which is below:

insignificantDigits = abs(vDecimal) % divisor

can we make this line of code more versatile please? 

One of our employees tagged the fact below:

    <us-gaap:IncomeLossFromDiscontinuedOperationsNetOfTaxPerBasicShare
      contextRef="C_a32cec8c-3b4f-4875-801f-2f8ff0368484"
      decimals="2"
      id="F_9e4fa528-f075-4f16-9685-e505a0ddfbb6"
      unitRef="U_UnitedStatesOfAmericaDollarsShare">388163667900000000000000000</us-gaap:IncomeLossFromDiscontinuedOperationsNetOfTaxPerBasicShare>

, which results in the following error below:

[exception:InvalidOperation] Instance validation exception: [<class 'decimal.DivisionImpossible'>], instance: mlm-20231231.htm - mlm-20231231.htm 
Traceback (most recent call last):
  File "D:\a\Arelle\Arelle\arelle\Validate.py", line 109, in validate
  File "D:\a\Arelle\Arelle\arelle\ValidateXbrl.py", line 401, in validate
  File "C:\Program Files\Arelle\plugin\validate\EFM\__init__.py", line 349, in validateXbrlFinally
    validateFiling(val, modelXbrl, isEFM=True)
  File "C:\Program Files\Arelle\plugin\validate\EFM\Filing.py", line 858, in validateFiling
    insignificance = insignificantDigits(f1.xValue, decimals=f1.decimals)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\Arelle\Arelle\arelle\ValidateXbrlCalcs.py", line 741, in insignificantDigits
decimal.InvalidOperation: [<class 'decimal.DivisionImpossible'>]

It took me several hours to find the cause of the error.

Discussion:
aviary-wf: ## Security Insights
No security relevant content was detected by automated scans.

## Action Items
* Review PR for [security impact](https://wiki.atl.workiva.net/display/SECURITY/Development+Security+Review+Guidelines); comment "security review required" if needed or unsure
* Verify [`aviary.yaml`](https://wiki.atl.workiva.net/pages/viewpage.action?pageId=42762752#SecurityInsightFramework&Tooling-ConfiguringAviary) coverage of security relevant code
---
_Questions or Comments? Reach out on Slack: #support-infosec._

Patch:
diff --git a/arelle/ValidateXbrlCalcs.py b/arelle/ValidateXbrlCalcs.py
index 068bcc36..519a150d 100644
--- a/arelle/ValidateXbrlCalcs.py
+++ b/arelle/ValidateXbrlCalcs.py
@@ -16,9 +16,11 @@ from arelle.XmlValidateConst import UNVALIDATED, VALID
 if TYPE_CHECKING:
     from _decimal import Decimal
     from arelle.ModelInstanceObject import ModelFact
+    from arelle.ModelValue import TypeXValue
 else:
     ModelFact = None # circular import with ModelInstanceObject
 
+
 def init(): # prevent circular imports
     global ModelFact
     if ModelFact is None:
@@ -694,55 +696,39 @@ def rangeValue(value, decimals=None, truncate=False) -> tuple[decimal.Decimal, d
                 return (vDecimal - dd, vDecimal + dd, False, False)
     return (vDecimal, vDecimal, True, True)
 
-def insignificantDigits(value, precision=None, decimals=None, scale=None) -> tuple[Decimal, Decimal] | None:
+
+def insignificantDigits(
+        value: TypeXValue,
+        decimals: int | float | Decimal | str) -> tuple[Decimal, Decimal] | None:
+    # Normalize value
     try:
-        vDecimal = decimal.Decimal(value)
-        if scale:
-            iScale = int(scale)
-            vDecimal = vDecimal.scaleb(iScale)
-        if precision is not None:
-            vFloat = float(value)
-            if scale:
-                vFloat = pow(vFloat, iScale)
-    except (decimal.InvalidOperation, ValueError): # would have been a schema error reported earlier
+        valueDecimal = decimal.Decimal(value)
+    except (decimal.InvalidOperation, ValueError):  # would have been a schema error reported earlier
         return None
-    if precision is not None:
-        if not isinstance(precision, (int,float)):
-            if precision == "INF":
-                return None
-            else:
-                try:
-                    precision = int(precision)
-                except ValueError: # would be a schema error
-                    return None
-        if isinf(precision) or precision == 0 or isnan(precision) or vFloat == 0:
+    if not valueDecimal.is_normal():  # prevent exception with excessive quantization digits
+        return None
+    # Normalize decimals
+    if isinstance(decimals, str):
+        if decimals == "INF":
             return None
         else:
-            vAbs = fabs(vFloat)
-            log = log10(vAbs)
-            decimals = precision - int(log) - (1 if vAbs >= 1 else 0)
-    elif decimals is not None:
-        if not isinstance(decimals, (int,float)):
-            if decimals == "INF":
+            try:
+                decimals = int(decimals)
+            except ValueError:  # would have been a schema error reported earlier
                 return None
-            else:
-                try:
-                    decimals = int(decimals)
-                except ValueError: # would be a schema error
-                    return None
-        if isinf(decimals) or isnan(decimals):
-            return None
-    else:
+    if isinf(decimals) or isnan(decimals) or decimals <= -28:  # prevent exception with excessive quantization digits
+        return None
+    if decimals > 0:
+        divisor = ONE.scaleb(-decimals)  # fractional scaling doesn't produce scientific notation
+    else:  # extra quantize step to prevent scientific notation for decimal number
+        divisor = ONE.scaleb(-decimals).quantize(ONE, decimal.ROUND_HALF_UP) # should never round
+    try:
+        quotient, insignificant = divmod(valueDecimal, divisor)
+    except decimal.InvalidOperation:
         return None
-    if vDecimal.is_normal() and -28 <= decimals <= 28: # prevent exception with excessive quantization digits
-        if decimals > 0:
-            divisor = ONE.scaleb(-decimals) # fractional scaling doesn't produce scientific notation
-        else:  # extra quantize step to prevent scientific notation for decimal number
-            divisor = ONE.scaleb(-decimals).quantize(ONE, decimal.ROUND_HALF_UP) # should never round
-        insignificantDigits = abs(vDecimal) % divisor
-        if insignificantDigits:
-            return (vDecimal // divisor * divisor,  # truncated portion of number
-                    insignificantDigits)   # nsignificant digits portion of number
+    if insignificant:
+        significant = quotient * divisor
+        return significant, abs(insignificant)
     return None
 
 


Test patch:
diff --git a/tests/unit_tests/arelle/test_validatexbrlcalcs.py b/tests/unit_tests/arelle/test_validatexbrlcalcs.py
index 7e8c9ea5..28b81b7c 100644
--- a/tests/unit_tests/arelle/test_validatexbrlcalcs.py
+++ b/tests/unit_tests/arelle/test_validatexbrlcalcs.py
@@ -5,162 +5,88 @@ from _decimal import Decimal, InvalidOperation
 from arelle.ValidateXbrlCalcs import insignificantDigits
 
 
-@pytest.mark.parametrize('value, precision, decimals, scale, result, error', [
-    # precision
-    ('1234', '-1', None, None, ('0', '1234'), None),
-    ('1234', '0', None, None, None, None),
-    ('1234', '1', None, None, ('1000', '234'), None),
-    ('1234', '2', None, None, ('1200', '34'), None),
-    ('1234', '3', None, None, ('1230', '4'), None),
-    ('1234', '4', None, None, None, None),
-    ('1234', '5', None, None, None, None),
-
-    ('1234.5678', '-2', None, None, ('0', '1234.5678'), None),
-    ('1234.5678', '-1', None, None, ('0', '1234.5678'), None),
-    ('1234.5678', '0', None, None, None, None),
-    ('1234.5678', '1', None, None, ('1000', '234.5678'), None),
-    ('1234.5678', '5', None, None, ('1234.5', '0.0678'), None),
-    ('1234.5678', '9', None, None, None, None),
-
+@pytest.mark.parametrize('value, decimals, result', [
     # decimals
-    ('1234', None, '-5', None, ('0', '1234'), None),
-    ('1234', None, '-1', None, ('1230', '4'), None),
-    ('1234', None, '0', None, None, None),
-    ('1234', None, '1', None, None, None),
-
-    ('1234.5678', None, '-5', None, ('0', '1234.5678'), None),
-    ('1234.5678', None, '-1', None, ('1230', '4.5678'), None),
-    ('1234.5678', None, '0', None, ('1234', '0.5678'), None),
-    ('1234.5678', None, '1', None, ('1234.5', '0.0678'), None),
-    ('1234.5678', None, '5', None, None, None),
-
-    # precision + scale
-    ('1234', '-1', None, '-1', None, None),
-    ('1234', '-1', None, '0', ('0', '1234'), None),
-    ('1234', '-1', None, '1', ('0', '12340'), None),
-    ('1234', '0', None, '-1', None, None),
-    ('1234', '0', None, '0', None, None),
-    ('1234', '0', None, '1', None, None),
-    ('1234', '1', None, '-5', None, None),
-    ('1234', '1', None, '-1', None, None),
-    ('1234', '1', None, '0', ('1000', '234'), None),
-    ('1234', '1', None, '1', ('12000', '340'), None),
-
-    ('1234.5678', '-1', None, '-1', ('123.45', '0.00678'), None),
-    ('1234.5678', '-1', None, '0', ('0', '1234.5678'), None),
-    ('1234.5678', '-1', None, '1', ('0', '12345.678'), None),
-    ('1234.5678', '0', None, '-1', None, None),
-    ('1234.5678', '0', None, '0', None, None),
-    ('1234.5678', '0', None, '1', None, None),
-    ('1234.5678', '1', None, '-1', ('123.4567', '0.00008'), None),
-    ('1234.5678', '1', None, '0', ('1000', '234.5678'), None),
-    ('1234.5678', '1', None, '1', ('12000', '345.678'), None),
-
-    # decimals + scale
-    ('1234', None, '-1', '-5', ('0', '0.01234'), None),
-    ('1234', None, '-1', '-1', ('120', '3.4'), None),
-    ('1234', None, '-1', '0', ('1230', '4'), None),
-    ('1234', None, '-1', '1', None, None),
-    ('1234', None, '-1', '2', None, None),
-    ('1234', None, '0', '-5', ('0', '0.01234'), None),
-    ('1234', None, '0', '-1', ('123', '0.4'), None),
-    ('1234', None, '0', '0', None, None),
-    ('1234', None, '0', '1', None, None),
-    ('1234', None, '1', '-5', ('0', '0.01234'), None),
-    ('1234', None, '1', '-1', None, None),
-    ('1234', None, '1', '0', None, None),
-    ('1234', None, '1', '1', None, None),
-
-    ('1234.5678', None, '-1', '-5', ('0', '0.012345678'), None),
-    ('1234.5678', None, '-1', '-1', ('120', '3.45678'), None),
-    ('1234.5678', None, '-1', '0', ('1230', '4.5678'), None),
-    ('1234.5678', None, '-1', '1', ('12340', '5.678'), None),
-    ('1234.5678', None, '0', '-5', ('0', '0.012345678'), None),
-    ('1234.5678', None, '0', '-1', ('123', '0.45678'), None),
-    ('1234.5678', None, '0', '0', ('1234', '0.5678'), None),
-    ('1234.5678', None, '0', '1', ('12345', '0.678'), None),
-    ('1234.5678', None, '1', '-5', ('0', '0.012345678'), None),
-    ('1234.5678', None, '1', '-1', ('123.4', '0.05678'), None),
-    ('1234.5678', None, '1', '0', ('1234.5', '0.0678'), None),
-    ('1234.5678', None, '1', '1', ('12345.6', '0.078'), None),
-
-    # large precision
-    ('1',  '1', None, None, None, None),
-    ('1', '27', None, None, None, None),
-    ('1', '28', None, None, None, None),
-    ('1', '29', None, None, None, InvalidOperation),
-    ('1', '30', None, None, None, None),
-
-    ('1',  '1', None, '1', None, None),
-    ('1', '27', None, '1', None, None),
-    ('1', '28', None, '1', None, InvalidOperation),
-    ('1', '29', None, '1', None, InvalidOperation),
-    ('1', '30', None, '1', None, None),
-
-    ('1',  '1', None, '2', None, None),
-    ('1', '27', None, '2', None, InvalidOperation),
-    ('1', '28', None, '2', None, InvalidOperation),
-    ('1', '29', None, '2', None, InvalidOperation),
-    ('1', '30', None, '2', None, None),
+    ('1234', '-5', ('0', '1234')),
+    ('1234', '-1', ('1230', '4')),
+    ('1234', '0', None),
+    ('1234', '1', None),
+    ('-1234', '-5', ('0', '1234')),
+    ('-1234', '-1', ('-1230', '4')),
+    ('-1234', '0', None),
+    ('-1234', '1', None),
+
+    ('1234.5678', '-5', ('0', '1234.5678')),
+    ('1234.5678', '-1', ('1230', '4.5678')),
+    ('1234.5678', '0', ('1234', '0.5678')),
+    ('1234.5678', '1', ('1234.5', '0.0678')),
+    ('1234.5678', '5', None),
+
+    ('-1234.5678', '-5', ('0', '1234.5678')),
+    ('-1234.5678', '-1', ('-1230', '4.5678')),
+    ('-1234.5678', '0', ('-1234', '0.5678')),
+    ('-1234.5678', '1', ('-1234.5', '0.0678')),
+    ('-1234.5678', '5', None),
 
     # large decimals
-    ('1', None, '27', None, None, None),
-    ('1', None, '28', None, None, InvalidOperation),
-    ('1', None, '29', None, None, None),
-
-    ('1', None, '26', '1', None, None),
-    ('1', None, '27', '1', None, InvalidOperation),
-    ('1', None, '28', '1', None, InvalidOperation),
-    ('1', None, '29', '1', None, None),
-
-    ('1', None, '25', '2', None, None),
-    ('1', None, '26', '2', None, InvalidOperation),
-    ('1', None, '27', '2', None, InvalidOperation),
-    ('1', None, '28', '2', None, InvalidOperation),
-    ('1', None, '29', '2', None, None),
+    ('1', '27', None),
+    ('1', '28', None),
+    ('1', '29', None),
+
+    ('1', '26', None),
+    ('1', '27', None),
+    ('1', '28', None),
+    ('1', '29', None),
+
+    ('1', '25', None),
+    ('1', '26', None),
+    ('1', '27', None),
+    ('1', '28', None),
+    ('1', '29', None),
+
+    ('1.1E26', '-26', ('1E26', '1E25')),
+    ('1.1E27', '-27', ('1E27', '1E26')),
+    ('1.1E28', '-28', None),  # 28 decimals too many for quantization
+    ('1.1E-27', '27', ('1E-27', '1E-28')),
+    ('1.1E-28', '28', ('1E-28', '1E-29')),
+    ('1.1E-29', '29', ('1E-29', '1E-30')),
+
+    ('-1.1E26', '-26', ('-1E26', '1E25')),
+    ('-1.1E27', '-27', ('-1E27', '1E26')),
+    ('-1.1E28', '-28', None),  # 28 decimals too many for quantization
+    ('-1.1E-27', '27', ('-1E-27', '1E-28')),
+    ('-1.1E-28', '28', ('-1E-28', '1E-29')),
+    ('-1.1E-29', '29', ('-1E-29', '1E-30')),
 
     # large whole values
-    ('1E27', None,  '0', None, None, None),
-    ('1E28', None,  '0', None, None, InvalidOperation),
-    ('1E26', None,  '1', None, None, None),
-    ('1E27', None,  '1', None, None, InvalidOperation),
+    ('1E27',  '0', None),
+    ('1E28',  '0', None),
+    ('1E26',  '1', None),
+    ('1E27',  '1', None),
 
     # large fractional values
-    ('1.1E27', None,  '0', None, None, None),
-    ('1.1E28', None,  '0', None, None, InvalidOperation),
-    ('1.1E26', None,  '1', None, None, None),
-    ('1.1E27', None,  '1', None, None, InvalidOperation),
-    ('123456789012345678901234567.1', None,  '0', None, ('123456789012345678901234567', '0.1'), None),
-    ('12345678901234567890123456789.1', None,  '0', None, None, InvalidOperation),
+    ('1.1E27', '0', None),
+    ('1.1E28', '0', None),
+    ('1.1E26', '1', None),
+    ('1.1E27', '1', None),
+    ('123456789012345678901234567.1', '0', ('123456789012345678901234567', '0.1')),
+    ('12345678901234567890123456789.1', '0', None),
 
     # small fractional values
-    ('1E-100', None,  '0', None, ('0', '1E-100'), None),
-    ('1.1E-100', None,  '0', None, ('0', '1.1E-100'), None),
-    ('0.1000000000000000000000000001', None,  '0', None, ('0', '0.1000000000000000000000000001'), None),
-    ('0.10000000000000000000000000001', None,  '0', None, ('0', '0.1'), None),
-    ('0.01000000000000000000000000001', None,  '0', '1', ('0', '0.1000000000000000000000000001'), None),
-    ('0.010000000000000000000000000001', None,  '0', '1', ('0', '0.1'), None),
+    ('1E-100', '0', ('0', '1E-100')),
+    ('1.1E-100', '0', ('0', '1.1E-100')),
+    ('0.1000000000000000000000000001', '0', ('0', '0.1000000000000000000000000001')),
+    ('0.10000000000000000000000000001', '0', ('0', '0.1')),
 ])
 def test_insignificantDigits(
         value: str,
-        precision: str | None,
-        decimals: str | None,
-        scale: str | None,
-        result: tuple[str, str] | None,
-        error: type | None) -> None:
+        decimals: str,
+        result: tuple[str, str] | None) -> None:
     expected_result = (Decimal(result[0]), Decimal(result[1])) \
         if isinstance(result, tuple) \
         else result
-    actual_error = None
-    actual_result = None
-    try:
-        actual_result = insignificantDigits(
-            Decimal(value) if value is not None else None,
-            Decimal(precision) if precision is not None else None,
-            Decimal(decimals) if decimals is not None else None,
-            Decimal(scale) if scale is not None else None
-        )
-    except Exception as exc:
-        actual_error = exc
-    assert (actual_error is None and error is None) or type(actual_error) == error
+    actual_result = insignificantDigits(
+        Decimal(value),
+        Decimal(decimals)
+    )
     assert actual_result == expected_result