Repository: numba/numba

Issue:
llvm_pass_timings fails to parse some raw data on 32-bit
I get 4 identical test failures when building numba 0.53 on the openSUSE build service for Tumbleweed on intel 32-bit ("i586"):

```
[   15s] + /usr/bin/numba-3.8 -s
[   16s] System info:
[   16s] --------------------------------------------------------------------------------
[   16s] __Time Stamp__
[   16s] Report started (local time)                   : 2021-03-16 22:32:52.817788
[   16s] UTC start time                                : 2021-03-16 22:32:52.817792
[   16s] Running time (s)                              : 0.32707
[   16s] 
[   16s] __Hardware Information__
[   16s] Machine                                       : i686
[   16s] CPU Name                                      : znver2
[   16s] CPU Count                                     : 12
[   16s] Number of accessible CPUs                     : 12
[   16s] List of accessible CPUs cores                 : 0-11
[   16s] CFS Restrictions (CPUs worth of runtime)      : None
[   16s] 
[   16s] CPU Features                                  : 64bit adx aes avx avx2 bmi bmi2
[   16s]                                                 clflushopt clwb clzero cmov cx16
[   16s]                                                 cx8 f16c fma fsgsbase fxsr lzcnt
[   16s]                                                 mmx movbe mwaitx pclmul popcnt
[   16s]                                                 prfchw rdpid rdrnd rdseed sahf sha
[   16s]                                                 sse sse2 sse3 sse4.1 sse4.2 sse4a
[   16s]                                                 ssse3 wbnoinvd xsave xsavec
[   16s]                                                 xsaveopt xsaves
[   16s] 
[   16s] Memory Total (MB)                             : 32009
[   16s] Memory Available (MB)                         : 27014
[   16s] 
[   16s] __OS Information__
[   16s] Platform Name                                 : Linux-5.11.4-1-default-i686-athlon-with-glibc2.1.3
[   16s] Platform Release                              : 5.11.4-1-default
[   16s] OS Name                                       : Linux
[   16s] OS Version                                    : #1 SMP Mon Mar 8 05:16:55 UTC 2021 (be77cd2)
[   16s] OS Specific Version                           : ?
[   16s] Libc Version                                  : glibc 2.33
[   16s] 
[   16s] __Python Information__
[   16s] Python Compiler                               : GCC
[   16s] Python Implementation                         : CPython
[   16s] Python Version                                : 3.8.8
[   16s] Python Locale                                 : en_US.UTF-8
[   16s] 
[   16s] __LLVM Information__
[   16s] LLVM Version                                  : 10.0.1
[   16s] 
[   16s] __CUDA Information__
[   16s] CUDA Device Initialized                       : False
[   16s] CUDA Driver Version                           : ?
[   16s] CUDA Detect Output:
[   16s] None
[   16s] CUDA Libraries Test Output:
[   16s] None
[   16s] 
[   16s] __ROC information__
[   16s] ROC Available                                 : False
[   16s] ROC Toolchains                                : None
[   16s] HSA Agents Count                              : 0
[   16s] HSA Agents:
[   16s] None
[   16s] HSA Discrete GPUs Count                       : 0
[   16s] HSA Discrete GPUs                             : None
[   16s] 
[   16s] __SVML Information__
[   16s] SVML State, config.USING_SVML                 : False
[   16s] SVML Library Loaded                           : False
[   16s] llvmlite Using SVML Patched LLVM              : False
[   16s] SVML Operational                              : False
[   16s] 
[   16s] __Threading Layer Information__
[   16s] TBB Threading Layer Available                 : True
[   16s] +-->TBB imported successfully.
[   16s] OpenMP Threading Layer Available              : True
[   16s] +-->Vendor: GNU
[   16s] Workqueue Threading Layer Available           : True
[   16s] +-->Workqueue imported successfully.
[   16s] 
[   16s] __Numba Environment Variable Information__
[   16s] NUMBA_PARALLEL_DIAGNOSTICS                    : 1
[   16s] 
[   16s] __Conda Information__
[   16s] Conda not available.
[   16s] 
[   16s] __Installed Packages__
[   16s] Package            Version
[   16s] ------------------ -------
[   16s] apipkg             1.5
[   16s] appdirs            1.4.4
[   16s] attrs              20.3.0
[   16s] Babel              2.9.0
[   16s] backcall           0.2.0
[   16s] cffi               1.14.5
[   16s] decorator          4.4.2
[   16s] importlib-metadata 3.7.0
[   16s] iniconfig          0.0.0
[   16s] ipython            7.20.0
[   16s] ipython-genutils   0.2.0
[   16s] jedi               0.17.2
[   16s] Jinja2             2.11.3
[   16s] llvmlite           0.36.0
[   16s] MarkupSafe         1.1.1
[   16s] more-itertools     8.7.0
[   16s] numba              0.53.0
[   16s] numpy              1.20.1
[   16s] ordered-set        3.1.1
[   16s] packaging          20.8
[   16s] parso              0.7.1
[   16s] pexpect            4.8.0
[   16s] pickleshare        0.7.5
[   16s] pip                20.2.4
[   16s] pluggy             0.13.1
[   16s] prompt-toolkit     3.0.8
[   16s] ptyprocess         0.7.0
[   16s] py                 1.10.0
[   16s] pybind11           2.6.2
[   16s] pycparser          2.20
[   16s] Pygments           2.7.4
[   16s] pyparsing          2.4.7
[   16s] pytest             6.2.2
[   16s] pytz               2021.1
[   16s] PyYAML             5.4.1
[   16s] scipy              1.6.1
[   16s] setuptools         44.1.1
[   16s] simplegeneric      0.8.1
[   16s] six                1.15.0
[   16s] TBB                2020.3
[   16s] toml               0.10.2
[   16s] traitlets          4.3.3
[   16s] wcwidth            0.2.5
[   16s] zipp               3.4.1
[   16s] 
[   16s] No errors reported.
[   16s] 
[   16s] 
[   16s] __Warning log__
[   16s] Warning (cuda): CUDA is disabled or no CUDA enabled devices are present.
[   16s] Exception class: <class 'numba.cuda.cudadrv.error.CudaSupportError'>
[   16s] Warning (roc): Error initialising ROC: No ROC toolchains found.
[   16s] Warning (roc): No HSA Agents found, encountered exception when searching: Error at driver init: 
[   16s] 
[   16s] HSA is not currently supported on this platform (linux).
[   16s] :
[   16s] Warning: Conda not available.
[   16s]  Error was [Errno 2] No such file or directory: 'conda'
[   16s] 
[   16s] Warning (psutil): psutil cannot be imported. For more accuracy, consider installing it.
[   16s] Warning (no file): /sys/fs/cgroup/cpuacct/cpu.cfs_quota_us
[   16s] Warning (no file): /sys/fs/cgroup/cpuacct/cpu.cfs_period_us
[   16s] --------------------------------------------------------------------------------
[   16s] If requested, please copy and paste the information between
[   16s] the dashed (----) lines, or from a given specific section as
[   16s] appropriate.
[   16s] 
[   16s] =============================================================
[   16s] IMPORTANT: Please ensure that you are happy with sharing the
[   16s] contents of the information present, any information that you
[   16s] wish to keep private you should remove before sharing.
[   16s] =============================================================
[   16s] 
[   16s] + python3.8 -m numba.runtests -v -b --exclude-tags=long_running -m 12 -- numba.tests
[   17s] skipped CUDA tests
[   17s] skipped CUDA tests
[   17s] skipped HSA tests
[   17s] skipped HSA tests
[   17s] Parallel: 8015. Serial: 606
...
[  491s] ======================================================================
[  491s] ERROR: test_pass_timings (numba.tests.doc_examples.test_llvm_pass_timings.DocsLLVMPassTimings)
[  491s] ----------------------------------------------------------------------
[  491s] Traceback (most recent call last):
[  491s]   File "/usr/lib/python3.8/site-packages/numba/tests/doc_examples/test_llvm_pass_timings.py", line 25, in test_pass_timings
[  491s]     print(md['llvm_pass_timings'])
[  491s]   File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 400, in __str__
[  491s]     return self.summary()
[  491s]   File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 372, in summary
[  491s]     overall_time = self.get_total_time()
[  491s]   File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 330, in get_total_time
[  491s]     return sum(r.timings.get_total_time() for r in self._records)
[  491s]   File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 330, in <genexpr>
[  491s]     return sum(r.timings.get_total_time() for r in self._records)
[  491s]   File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 133, in get_total_time
[  491s]     return self.list_records()[-1].wall_time
[  491s]   File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 142, in list_records
[  491s]     return self._processed
[  491s]   File "/usr/lib/python3.8/site-packages/numba/core/utils.py", line 277, in __get__
[  491s]     res = instance.__dict__[self.name] = self.func(instance)
[  491s]   File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 195, in _processed
[  491s]     return self._process()
[  491s]   File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 266, in _process
[  491s]     return _adjust_timings(records)
[  491s]   File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 69, in _adjust_timings
[  491s]     total_rec = records[-1]
[  491s] IndexError: list index out of range
[  491s] 
[  491s] ======================================================================
[  491s] ERROR: test_timing_properties (numba.tests.test_event.TestEvent)
[  491s] ----------------------------------------------------------------------
[  491s] Traceback (most recent call last):
[  491s]   File "/usr/lib/python3.8/site-packages/numba/tests/test_event.py", line 185, in test_timing_properties
[  491s]     self.assertLess(foo_llvm_timer.get_total_time(),
[  491s]   File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 330, in get_total_time
...
[  491s]   File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 69, in _adjust_timings
[  491s]     total_rec = records[-1]
[  491s] IndexError: list index out of range
[  491s] 
[  491s] ======================================================================
[  491s] ERROR: test_analyze (numba.tests.test_llvm_pass_timings.TestLLVMPassTimings)
[  491s] ----------------------------------------------------------------------
[  491s] Traceback (most recent call last):
[  491s]   File "/usr/lib/python3.8/site-packages/numba/tests/test_llvm_pass_timings.py", line 52, in test_analyze
[  491s]     self.assertIsInstance(timings_collection.get_total_time(), float)
[  491s]   File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 330, in get_total_time
...
[  491s]   File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 69, in _adjust_timings
[  491s]     total_rec = records[-1]
[  491s] IndexError: list index out of range
[  491s] 
[  491s] ======================================================================
[  491s] ERROR: test_usage (numba.tests.test_llvm_pass_timings.TestLLVMPassTimings)
[  491s] ----------------------------------------------------------------------
[  491s] Traceback (most recent call last):
[  491s]   File "/usr/lib/python3.8/site-packages/numba/tests/test_llvm_pass_timings.py", line 26, in test_usage
[  491s]     text = str(timings)
...
[  491s]   File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 69, in _adjust_timings
[  491s]     total_rec = records[-1]
[  491s] IndexError: list index out of range
[  491s] 
[  491s] ----------------------------------------------------------------------
[  491s] Ran 8621 tests in 473.795s
[  491s] 
[  491s] FAILED (errors=4, skipped=533, expected failures=12)

```


The offending raw data derived from the code of `test_usage` is:
```
abuild@skylab:~> python3
Python 3.8.8 (default, Feb 19 2021, 16:53:21) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import unittest
>>> 
>>> from numba import njit
>>> from numba.tests.support import TestCase, override_config
>>> from numba.misc import llvm_pass_timings as lpt
>>> @njit
... def foo(n): 
...             c = 0
...             for i in range(n):
...                 c += i
...             return c
>>> with override_config('LLVM_PASS_TIMINGS', True):
...             foo(10)                     
... 
45
>>> md = foo.get_metadata(foo.signatures[0])
>>> timings = md['llvm_pass_timings']
>>> str(timings)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 400, in __str__
    return self.summary()
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 372, in summary
    overall_time = self.get_total_time()
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 330, in get_total_time
    return sum(r.timings.get_total_time() for r in self._records)
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 330, in <genexpr>
    return sum(r.timings.get_total_time() for r in self._records)
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 133, in get_total_time
    return self.list_records()[-1].wall_time
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 142, in list_records
    return self._processed
  File "/usr/lib/python3.8/site-packages/numba/core/utils.py", line 277, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 195, in _processed
    return self._process()
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 266, in _process
    return _adjust_timings(records)
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 69, in _adjust_timings
    total_rec = records[-1]
IndexError: list index out of range
>>> timings._records[0].timings
<numba.misc.llvm_pass_timings.ProcessedPassTimings object at 0xdcb93118>
>>> timings._records[0].timings.get_total_time()
0.0007
>>> timings._records[1].timings.get_total_time()
0.0002
>>> timings._records[2].timings.get_total_time()
0.0001
>>> timings._records[3].timings.get_total_time()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 133, in get_total_time
    return self.list_records()[-1].wall_time
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 142, in list_records
    return self._processed
  File "/usr/lib/python3.8/site-packages/numba/core/utils.py", line 277, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 195, in _processed
    return self._process()
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 266, in _process
    return _adjust_timings(records)
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 69, in _adjust_timings
    total_rec = records[-1]
IndexError: list index out of range
>>> timings._records[3].timings._process()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 266, in _process
    return _adjust_timings(records)
  File "/usr/lib/python3.8/site-packages/numba/misc/llvm_pass_timings.py", line 69, in _adjust_timings
    total_rec = records[-1]
IndexError: list index out of range
>>> print(timings._records[3].timings._raw_data)
===-------------------------------------------------------------------------===
                      ... Pass execution timing report ...
===-------------------------------------------------------------------------===
  Total Execution Time: 0.0007 seconds (0.0007 wall clock)

   ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
   0.0004 ( 55.5%)        -----        0.0004 ( 55.5%)   0.0004 ( 56.0%)  Combine redundant instructions
   0.0001 (  8.7%)        -----        0.0001 (  8.7%)   0.0001 (  8.8%)  Prune NRT refops #4
   0.0000 (  6.9%)        -----        0.0000 (  6.9%)   0.0000 (  5.7%)  Function Integration/Inlining
   0.0000 (  3.8%)        -----        0.0000 (  3.8%)   0.0000 (  3.9%)  Post-Dominator Tree Construction #4
   0.0000 (  2.9%)        -----        0.0000 (  2.9%)   0.0000 (  2.9%)  Natural Loop Information
   0.0000 (  2.7%)        -----        0.0000 (  2.7%)   0.0000 (  2.6%)  Dominator Tree Construction #8
   0.0000 (  2.4%)        -----        0.0000 (  2.4%)   0.0000 (  2.6%)  Dominator Tree Construction #7
   0.0000 (  2.5%)        -----        0.0000 (  2.5%)   0.0000 (  2.3%)  Normalize NRT refops #4
   0.0000 (  2.2%)        -----        0.0000 (  2.2%)   0.0000 (  2.3%)  CallGraph Construction
   0.0000 (  1.8%)        -----        0.0000 (  1.8%)   0.0000 (  1.8%)  Rotate Loops
   0.0000 (  1.7%)        -----        0.0000 (  1.7%)   0.0000 (  1.6%)  Scalar Evolution Analysis
   0.0000 (  1.8%)        -----        0.0000 (  1.8%)   0.0000 (  1.5%)  Canonicalize natural loops
   0.0000 (  1.1%)        -----        0.0000 (  1.1%)   0.0000 (  1.4%)  Assumption Cache Tracker
   0.0000 (  1.0%)        -----        0.0000 (  1.0%)   0.0000 (  1.3%)  Loop-Closed SSA Form Pass
   0.0000 (  1.1%)        -----        0.0000 (  1.1%)   0.0000 (  1.0%)  Target Library Information
   0.0000 (  0.8%)        -----        0.0000 (  0.8%)   0.0000 (  1.0%)  Profile summary info
   0.0000 (  0.7%)        -----        0.0000 (  0.7%)   0.0000 (  0.9%)  Function Alias Analysis Results
   0.0000 (  0.7%)        -----        0.0000 (  0.7%)   0.0000 (  0.6%)  Basic Alias Analysis (stateless AA impl)
   0.0000 (  0.7%)        -----        0.0000 (  0.7%)   0.0000 (  0.5%)  Lazy Branch Probability Analysis
   0.0000 (  0.1%)        -----        0.0000 (  0.1%)   0.0000 (  0.4%)  Lazy Block Frequency Analysis
   0.0000 (  0.3%)        -----        0.0000 (  0.3%)   0.0000 (  0.4%)  Optimization Remark Emitter
   0.0000 (  0.1%)        -----        0.0000 (  0.1%)   0.0000 (  0.3%)  LCSSA Verifier
   0.0000 (  0.1%)        -----        0.0000 (  0.1%)   0.0000 (  0.3%)  Force set function attributes
   0.0000 (  0.1%)        -----        0.0000 (  0.1%)   0.0000 (  0.1%)  Target Transform Information
   0.0007 (100.0%)        -----        0.0007 (100.0%)   0.0007 (100.0%)  Total

```

My guess the problem are the missing values for the `System Time` column.

The failures do not happen on x86_64 or ARM architectures.

- [x] I have tried using the latest released version of Numba (most recent is
 visible in the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
- [x] I have included a self contained code sample to reproduce the problem.
  i.e. it's possible to run as 'python bug.py'.



Discussion:
Thanks for the report and for debugging this @bnavigator, on inspection, I think your assessment is correct. I wonder why the 32 bit builds in the build farm/public CI didn't pick this up. Maybe it's something to do with the way LLVM interacts with the OS/the OS itself and 32 bit is a red herring, will investigate. This is also a candidate for 0.53.1.

Resolving patch:
diff --git a/numba/misc/llvm_pass_timings.py b/numba/misc/llvm_pass_timings.py
--- a/numba/misc/llvm_pass_timings.py
+++ b/numba/misc/llvm_pass_timings.py
@@ -239,12 +239,14 @@ def parse(raw_data):
                     missing[k] = 0.0
             # parse timings
             n = r"\s*((?:[0-9]+\.)?[0-9]+)"
-            pat = f"\\s+{n}\\s*\\({n}%\\)" * (len(headers) - 1) + r"\s*(.*)"
+            pat = f"\\s+(?:{n}\\s*\\({n}%\\)|-+)" * (len(headers) - 1)
+            pat += r"\s*(.*)"
             for ln in line_iter:
                 m = re.match(pat, ln)
                 if m is not None:
                     raw_data = list(m.groups())
-                    data = {k: float(v) for k, v in zip(attrs, raw_data)}
+                    data = {k: float(v) if v is not None else 0.0
+                            for k, v in zip(attrs, raw_data)}
                     data.update(missing)
                     pass_name = raw_data[-1]
                     rec = PassTimingRecord(


Test patch:
diff --git a/numba/tests/test_llvm_pass_timings.py b/numba/tests/test_llvm_pass_timings.py
--- a/numba/tests/test_llvm_pass_timings.py
+++ b/numba/tests/test_llvm_pass_timings.py
@@ -5,6 +5,33 @@
 from numba.misc import llvm_pass_timings as lpt
 
 
+timings_raw1 = """
+===-------------------------------------------------------------------------===
+                      ... Pass execution timing report ...
+===-------------------------------------------------------------------------===
+  Total Execution Time: 0.0001 seconds (0.0001 wall clock)
+
+   ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
+   0.0001 ( 90.1%)   0.0001 ( 90.1%)   0.0001 ( 90.1%)   0.0001 ( 90.1%)  A1
+   0.0000 (  9.9%)   0.0000 (  9.9%)   0.0000 (  9.9%)   0.0000 (  9.9%)  A2
+   0.0001 (100.0%)   0.0001 (100.0%)   0.0001 (100.0%)   0.0001 (100.0%)  Total
+
+"""    # noqa: E501
+
+timings_raw2 = """
+===-------------------------------------------------------------------------===
+                      ... Pass execution timing report ...
+===-------------------------------------------------------------------------===
+  Total Execution Time: 0.0001 seconds (0.0001 wall clock)
+
+   ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
+   0.0001 ( 90.1%)        -----        0.0001 ( 90.1%)   0.0001 ( 90.1%)  A1
+   0.0000 (  9.9%)        -----        0.0000 (  9.9%)   0.0000 (  9.9%)  A2
+   0.0001 (100.0%)        -----        0.0001 (100.0%)   0.0001 (100.0%)  Total
+
+"""    # noqa: E501
+
+
 class TestLLVMPassTimings(TestCase):
 
     def test_usage(self):
@@ -61,6 +88,15 @@ def foo(n):
             self.assertGreaterEqual(last, cur)
             cur = last
 
+    def test_parse_raw(self):
+        timings1 = lpt.ProcessedPassTimings(timings_raw1)
+        self.assertAlmostEqual(timings1.get_total_time(), 0.0001)
+        self.assertIsInstance(timings1.summary(), str)
+
+        timings2 = lpt.ProcessedPassTimings(timings_raw2)
+        self.assertAlmostEqual(timings2.get_total_time(), 0.0001)
+        self.assertIsInstance(timings2.summary(), str)
+
 
 class TestLLVMPassTimingsDisabled(TestCase):
     def test_disabled_behavior(self):