Repository: scikit-hep/pyhf

Issue:
Order of pdf.confg.samples not guaranteed (due to use of set)
# Description

The order of samples in [`pdf.config.samples`](https://github.com/diana-hep/pyhf/blob/4fa45dda8d2314138549ac6b67ebdc9d9a506742/pyhf/pdf.py#L163) (more specifically, [`pdf._ModelConfig.samples`](https://github.com/diana-hep/pyhf/blob/5ddfc6aa21d1f2e33a2fe95bfefb489541ae77ff/pyhf/pdf.py#L35)) should not change between runs with the same data. However, it can, which is highly problematic when trying to do something like plotting as we do in the examples. If `pdf.config.samples` does not maintain order between runs, then something like [`get_mc_counts`](https://github.com/diana-hep/pyhf/blob/4fa45dda8d2314138549ac6b67ebdc9d9a506742/docs/examples/notebooks/binderexample/StatisticalAnalysis.ipynb) will give different results on different runs because `pdf._modifications(pars)` will be giving different results.

I think this is coming from the fact that the use of `set` in

https://github.com/diana-hep/pyhf/blob/5ddfc6aa21d1f2e33a2fe95bfefb489541ae77ff/pyhf/pdf.py#L93

[doesn't guarantee order](https://stackoverflow.com/questions/9792664/converting-a-list-to-a-set-changes-element-order).

I think what should be used instead is a sorted set

```python
self.samples = sorted(set(self.samples), key=self.samples.index)
```

Consider the following two scripts:

`bug_report.py`
```python
#!/usr/bin/env python3
import pyhf


def main():
    pdf = pyhf.simplemodels.hepdata_like(
        signal_data=[12.0, 11.0], bkg_data=[50.0, 52.0], bkg_uncerts=[3.0, 7.0]
    )
    print(pdf.config.samples)


if __name__ == '__main__':
    main()
```

and `bug_resport.sh`
```shell
#!/usr/bin/env bash

for i in {1..10}; do
   python3 bug_report.py 
done
```

Thanks to @stephensekula for uncovering some of this behavior.

# Expected Behavior

Running `bash bug_report.sh` should return

```
$ bash bug_report.sh 
['background', 'signal']
['background', 'signal']
['background', 'signal']
['background', 'signal']
['background', 'signal']
['background', 'signal']
['background', 'signal']
['background', 'signal']
['background', 'signal']
['background', 'signal']
```

# Actual Behavior

```
$ bash bug_report.sh 
['background', 'signal']
['signal', 'background']
['signal', 'background']
['background', 'signal']
['signal', 'background']
['signal', 'background']
['signal', 'background']
['signal', 'background']
['background', 'signal']
['signal', 'background']
```

# Steps to Reproduce

This Issue.

# Checklist

- [x] Run `git fetch` to get the most up to date version of `master`
- [x] Searched through existing Issues to confirm this is not a duplicate issue
- [x] Filled out the Description, Expected Behavior, Actual Behavior, and Steps to Reproduce sections above or have edited/removed them in a way that fully describes the issue



Discussion:
thanks @matthewfeickert -- just to double check: this is mostly about presentation, there is no internal inconsistency in the computation observed? I agree that having a deterministic order would be helpful.
> this is mostly about presentation, there is no internal inconsistency in the computation observed?

Correct. However, I think that I would call it interpretability rather then presentation. In the use case where this was first found this was manifested through the plotting order &mdash; and so rather noticeable &mdash; but if a user is accessing the samples for some further calculation this could be quite badly misleading if they believe that the order they got in 1 run will persist.

I need to look more at using sorted set as using

```python
self.samples = sorted(set(self.samples), key=self.samples.index)
```

does indeed fix the issue. However, there is any intent to the insertion order to the list then perhaps it would be better to use an `OrderedDict` which would definitely preserve the insertion order.

Resolving patch:
diff --git a/pyhf/pdf.py b/pyhf/pdf.py
--- a/pyhf/pdf.py
+++ b/pyhf/pdf.py
@@ -89,10 +89,10 @@ def __init__(self, spec, poiname='mu'):
                         paramset_requirements
                     )
 
-        self.channels = list(set(self.channels))
-        self.samples = list(set(self.samples))
-        self.parameters = list(set(self.parameters))
-        self.modifiers = list(set(self.modifiers))
+        self.channels = sorted(list(set(self.channels)))
+        self.samples = sorted(list(set(self.samples)))
+        self.parameters = sorted(list(set(self.parameters)))
+        self.modifiers = sorted(list(set(self.modifiers)))
         self.channel_nbins = self.channel_nbins
         self._create_and_register_paramsets(
             _paramsets_requirements, _paramsets_user_configs


Test patch: