Repository: HEPData/hepdata_lib

Issue:
Support inhomogeneous uncertainties
> I'm currently struggling to use the `hepdata_lib` for cases with inhomogeneous error breakdowns across bins. For instance, I have a distribution with three bins where the first two bins have two error components 'A' and 'B' (but not 'C') and the third bin has error component 'C' (but not 'A' and 'B').
>
> I know this is supported in principle, e.g. by just omitting the respective components in the dictionary. However, when using the library, it seems `hepdata_lib/helpers.py` raises a `ValueError` 
>
> ```
> ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (3,) + inhomogeneous part.
> ```
> Is there a trick?

_Originally posted by @20DM in https://github.com/HEPData/hepdata_lib/issues/229#issuecomment-2242550182_


> PS - just to be clear: of course I can "make it pass" by just setting the uncertainty to zero, but then all bins will have three uncertainty components, some of them zero, which is not the same as the bin not having the component in its breakdown to begin with. I think the problem is that the check for "non-zero uncertainties" only checks if there's at least one non-zero component and then adds all of them, regardless of their value. Can we make this more flexible?

_Originally posted by @20DM in https://github.com/HEPData/hepdata_lib/issues/229#issuecomment-2242610663_

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/HEPData/hepdata_lib/pull/265?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData) Report
Attention: Patch coverage is `50.00000%` with `2 lines` in your changes missing coverage. Please review.
> Project coverage is 90.22%. Comparing base [(`ba9f2e2`)](https://app.codecov.io/gh/HEPData/hepdata_lib/commit/ba9f2e28e33e59bb2bf5b429cab97d31cd556538?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData) to head [(`81adb42`)](https://app.codecov.io/gh/HEPData/hepdata_lib/commit/81adb429876820762c6b8608b9f36cc69d1040e9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData).

| [Files](https://app.codecov.io/gh/HEPData/hepdata_lib/pull/265?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData) | Patch % | Lines |
|---|---|---|
| [hepdata\_lib/\_\_init\_\_.py](https://app.codecov.io/gh/HEPData/hepdata_lib/pull/265?src=pr&el=tree&filepath=hepdata_lib%2F__init__.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData#diff-aGVwZGF0YV9saWIvX19pbml0X18ucHk=) | 0.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/HEPData/hepdata_lib/pull/265?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #265      +/-   ##
==========================================
- Coverage   90.37%   90.22%   -0.16%     
==========================================
  Files           5        5              
  Lines        1112     1115       +3     
  Branches      251      235      -16     
==========================================
+ Hits         1005     1006       +1     
- Misses         78       79       +1     
- Partials       29       30       +1     
```

| [Flag](https://app.codecov.io/gh/HEPData/hepdata_lib/pull/265/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData) | Coverage Δ | |
|---|---|---|
| [unittests-3.10](https://app.codecov.io/gh/HEPData/hepdata_lib/pull/265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData) | `90.22% <50.00%> (-0.16%)` | :arrow_down: |
| [unittests-3.11](https://app.codecov.io/gh/HEPData/hepdata_lib/pull/265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData) | `?` | |
| [unittests-3.6](https://app.codecov.io/gh/HEPData/hepdata_lib/pull/265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData) | `?` | |
| [unittests-3.7](https://app.codecov.io/gh/HEPData/hepdata_lib/pull/265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData) | `89.94% <50.00%> (-0.16%)` | :arrow_down: |
| [unittests-3.8](https://app.codecov.io/gh/HEPData/hepdata_lib/pull/265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData) | `90.04% <50.00%> (-0.16%)` | :arrow_down: |
| [unittests-3.9](https://app.codecov.io/gh/HEPData/hepdata_lib/pull/265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData) | `90.04% <50.00%> (-0.16%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/HEPData/hepdata_lib/pull/265?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=HEPData).

GraemeWatt: Thanks for adding the test and expanding the docs.  The CI is failing due to issues with `pylint`.  Can you please modify the code so that the `pylint` checks pass?  See [Analysing the code](https://hepdata-lib.readthedocs.io/en/latest/dev.html#analysing-the-code) in the developer docs.
20DM: > Thanks for fixing the linting for the tests. However, the CI still gives a message:
> 
> ```
> ************* Module hepdata_lib
> hepdata_lib/__init__.py:237:4: R0912: Too many branches (13/12) (too-many-branches)
> 
> -----------------------------------
> Your code has been rated at 9.99/10
> ```
> 
> Could you please fix this too? If you don't see an easy way to refactor the `make_dict` function to avoid this warning, it can be suppressed by adding a line `# pylint: disable=too-many-branches` at the top of the function, but this should be avoided if possible. I think it's a current bug in the CI that the "Run pylint" still passes, so I've opened a separate PR #267 to address it.

Urgh - well, this can be circumvented by moving the additional if statement into a separate list comprehensions, e.g. like so
```
for unc in [ unc for unc in self.uncertainties if not unc.values[i] is None ]:
    ...
```
but that adds an extra loop for no good reason. So my preference would be to silence the linter here, but I'd leave that call to you. 🤷 

Patch:
diff --git a/docs/usage.rst b/docs/usage.rst
index 410f000..31a183d 100644
--- a/docs/usage.rst
+++ b/docs/usage.rst
@@ -370,5 +370,7 @@ multiple dependent variables and a (different) subset of the bins has missing co
 In this case the uncertainties should be set to zero for the missing bins with a non-numeric central value like ``'-'``.
 The warning message can be suppressed by passing an optional argument ``zero_uncertainties_warning=False`` when
 defining an instance of the ``Variable`` class.
+Furthermore, note that `None` can be used to suppress the uncertainty for individual bins in cases where
+the uncertainty components may only apply to a subset of the values.
 
-.. _`Uncertainties`: https://hepdata-submission.readthedocs.io/en/latest/data_yaml.html#uncertainties
\ No newline at end of file
+.. _`Uncertainties`: https://hepdata-submission.readthedocs.io/en/latest/data_yaml.html#uncertainties
diff --git a/hepdata_lib/__init__.py b/hepdata_lib/__init__.py
index e919b5f..47282b6 100644
--- a/hepdata_lib/__init__.py
+++ b/hepdata_lib/__init__.py
@@ -235,6 +235,7 @@ class Variable:
         self.uncertainties.append(uncertainty)
 
     def make_dict(self):
+        # pylint: disable=too-many-branches
         """
         Return all data in this Variable as a dictionary.
 
@@ -277,6 +278,8 @@ class Variable:
             # if at least one of the uncertainties is not zero.
             if nonzero_uncs[i]:
                 for unc in self.uncertainties:
+                    if unc.values[i] is None:
+                        continue
                     if unc.is_symmetric:
                         valuedict['errors'].append({
                             "symerror":
diff --git a/hepdata_lib/helpers.py b/hepdata_lib/helpers.py
index e1df117..9e35973 100644
--- a/hepdata_lib/helpers.py
+++ b/hepdata_lib/helpers.py
@@ -258,7 +258,8 @@ def any_uncertainties_nonzero(uncertainties, size):
     for unc in uncertainties:
 
         # Treat one-sided uncertainties as
-        values = np.array(unc.values)
+        tmp = 0 if unc.is_symmetric else (0,0)
+        values = np.array([tmp if v is None else v for v in unc.values])
         values[values.astype(str)==''] = 0
         values = values.astype(float)
 
@@ -273,15 +274,17 @@ def sanitize_value(value):
     Handle conversion of input types for internal storage.
 
     :param value: User-side input value to sanitize.
-    :type value: string, int, or castable to float
+    :type value: string, int, NoneType, or castable to float
 
-    Strings and integers are left alone,
+    Strings, integers and None are left alone,
     everything else is converted to float.
     """
     if isinstance(value,str):
         return value
     if isinstance(value,int):
         return value
+    if value is None:
+        return value
     return float(value)
 
 


Test patch:
diff --git a/tests/test_uncertainty.py b/tests/test_uncertainty.py
index e121e3b..b982c76 100644
--- a/tests/test_uncertainty.py
+++ b/tests/test_uncertainty.py
@@ -115,3 +115,18 @@ class TestUncertainty(TestCase):
         # Check that 'errors' key is missing only if zero uncertainties
         self.assertTrue(all('errors' in dictionary['values'][i] for i in [0, 1, 3]))
         self.assertTrue('errors' not in dictionary['values'][2])
+
+    def test_inhomogenous_uncertainties(self):
+        '''Test cases where an uncertainty only applies to a subset of the bins'''
+        var = Variable("testvar", is_independent=False, is_binned=False, values=[1,2,3],
+                       zero_uncertainties_warning=False)
+        unc_a = Uncertainty('errorA', is_symmetric=True)
+        unc_a.values = [ 0.1, 0.2, None ]
+        var.add_uncertainty(unc_a)
+        unc_b = Uncertainty('errorB', is_symmetric=True)
+        unc_b.values = [ 0.1, 0.2, 0.3 ]
+        var.add_uncertainty(unc_b)
+        dictionary = var.make_dict()
+        self.assertTrue(len([ errs['label'] for i in [0,1,2] \
+                                            for errs in dictionary['values'][i]['errors'] \
+                                            if errs['label'] == 'errorA'])==2)