Repository: reframe-hpc/reframe

Issue:
Parameters values for filelog perflogs
Feel free to tell me that you don't want to support this use case if it's not useful for your sites.

We have a unit test that executes on each IB NIC/HCA on a node, this is done through a parameterized test:
```python
hca = parameter(["mlx5_0", "mlx5_1", "mlx5_2", "mlx5_3", "mlx5_6", "mlx5_7", "mlx5_8", "mlx5_9"])
```
With the naming scheme in 3.12.0, we could simply access the historical performance for one HCA given that the path was predictable:
```
$ cat ./logs/node0042/perflogs/ib_write_bw_loopback_mlx5_6.log
2022-08-07T09:54:13|reframe 3.12.0|ib_write_bw_loopback %hca=mlx5_6 ....
2022-08-08T10:28:11|reframe 3.12.0|ib_write_bw_loopback %hca=mlx5_6 ....
```
With 833c6ea3582bdcf4cceb6dc7a1aa667d0ca029e2, it's now a little more complex as the output file name is not predictable anymore, we would have to grep across all logs for this test to find the right HCA.

I understand the limitations of the old naming scheme mentioned in https://github.com/reframe-hpc/reframe/blob/833c6ea3582bdcf4cceb6dc7a1aa667d0ca029e2/docs/manpage.rst#test-naming-scheme, but perhaps we could have a way to modify the name of the `filelog` output file (doesn't seem possible today: https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#the-filelog-log-handler)? And then a new log record attribute that would allow to get parameters names in the file name.


Discussion:

Patch:
diff --git a/docs/config_reference.rst b/docs/config_reference.rst
index 7e70e4f3..9e1b6d35 100644
--- a/docs/config_reference.rst
+++ b/docs/config_reference.rst
@@ -1007,9 +1007,9 @@ The additional properties for the ``filelog`` handler are the following:
      {basedir}/
         system1/
             partition1/
-                test_name.log
+                test_short_name.log
             partition2/
-                test_name.log
+                test_short_name.log
             ...
         system2/
         ...
diff --git a/docs/manpage.rst b/docs/manpage.rst
index 8ee59874..e4fbad1a 100644
--- a/docs/manpage.rst
+++ b/docs/manpage.rst
@@ -125,6 +125,8 @@ This happens recursively so that if test ``T1`` depends on ``T2`` and ``T2`` dep
 
    If the special notation ``<test_name>@<variant_num>`` is passed as the ``NAME`` argument, then an exact match will be performed selecting the variant ``variant_num`` of the test ``test_name``.
 
+   You may also select a test by its hash code using the notation ``/<test-hash>`` for the ``NAME`` argument.
+
    .. note::
 
       Fixtures cannot be selected.
@@ -133,6 +135,11 @@ This happens recursively so that if test ``T1`` depends on ``T2`` and ``T2`` dep
 
       The option's behaviour was adapted and extended in order to work with the updated test naming scheme.
 
+   .. versionchanged:: 4.0.0
+
+      Support selecting tests by their hash code.
+
+
 .. option:: -p, --prgenv=NAME
 
    Filter tests by programming environment.
@@ -953,26 +960,26 @@ Here is how this test is listed where the various components of the display name
 
 .. code-block:: console
 
-   - TestA %x=4 %l.foo=10 %t.p=2
-       ^MyFixture %p=1 ~TestA_4_1
-       ^MyFixture %p=2 ~TestA_4_1
-       ^X %foo=10 ~generic:default+builtin
-   - TestA %x=3 %l.foo=10 %t.p=2
-       ^MyFixture %p=1 ~TestA_3_1
-       ^MyFixture %p=2 ~TestA_3_1
-       ^X %foo=10 ~generic:default+builtin
-   - TestA %x=4 %l.foo=10 %t.p=1
-       ^MyFixture %p=2 ~TestA_4_0
-       ^MyFixture %p=1 ~TestA_4_0
-       ^X %foo=10 ~generic:default+builtin
-   - TestA %x=3 %l.foo=10 %t.p=1
-       ^MyFixture %p=2 ~TestA_3_0
-       ^MyFixture %p=1 ~TestA_3_0
-       ^X %foo=10 ~generic:default+builtin
+   - TestA %x=4 %l.foo=10 %t.p=2 /1c51609b
+       ^Myfixture %p=1 ~TestA_3 /f027ee75
+       ^MyFixture %p=2 ~TestA_3 /830323a4
+       ^X %foo=10 ~generic:default+builtin /7dae3cc5
+   - TestA %x=3 %l.foo=10 %t.p=2 /707b752c
+       ^MyFixture %p=1 ~TestA_2 /02368516
+       ^MyFixture %p=2 ~TestA_2 /854b99b5
+       ^X %foo=10 ~generic:default+builtin /7dae3cc5
+   - TestA %x=4 %l.foo=10 %t.p=1 /c65657d5
+       ^MyFixture %p=2 ~TestA_1 /f0383f7f
+       ^MyFixture %p=1 ~TestA_1 /d07f4281
+       ^X %foo=10 ~generic:default+builtin /7dae3cc5
+   - TestA %x=3 %l.foo=10 %t.p=1 /1b9f44df
+       ^MyFixture %p=2 ~TestA_0 /b894ab05
+       ^MyFixture %p=1 ~TestA_0 /ca376ca8
+       ^X %foo=10 ~generic:default+builtin /7dae3cc5
    Found 4 check(s)
 
 Display names may not always be unique.
-In the following example:
+Assume the following test:
 
 .. code-block:: python
 
@@ -982,6 +989,19 @@ In the following example:
 This generates three different tests with different unique names, but their display name is the same for all: ``MyTest %p=1``.
 Notice that this example leads to a name conflict with the old naming scheme, since all tests would be named ``MyTest_1``.
 
+Each test is also associated with a hash code that is derived from the test name, its parameters and their values.
+As in the example listing above, the hash code of each test is printed with the :option:`-l` option and individual tests can be selected by their hash using the :option:`-n` option, e.g., ``-n /1c51609b``.
+The stage and output directories, as well as the performance log file of the ``filelog`` `performance log handler <config_reference.html#the-filelog-log-handler>`__ will use the hash code for the test-specific directories and files.
+This might lead to conflicts for tests as the one above when executing them with the asynchronous execution policy, but ensures consistency of performance record files when parameter values are added to or deleted from a test parameter.
+More specifically, the test's hash will not change if a new parameter value is added or deleted or even if the parameter values are shuffled.
+Test variants on the other side are more volatile and can change with such changes.
+Also users should not rely on how the variant numbers are assigned to a test, as this is an implementation detail.
+
+
+.. versionchanged:: 4.0.0
+
+   A hash code is associated with each test.
+
 
 --------------------------------------
 Differences from the old naming scheme
diff --git a/reframe/core/logging.py b/reframe/core/logging.py
index 25e7d014..4dc55193 100644
--- a/reframe/core/logging.py
+++ b/reframe/core/logging.py
@@ -147,8 +147,9 @@ class MultiFileHandler(logging.FileHandler):
         except OSError as e:
             raise LoggingError('logging failed') from e
 
-        self.baseFilename = os.path.join(dirname,
-                                         f'{record.__rfm_check__.name}.log')
+        self.baseFilename = os.path.join(
+            dirname, f'{record.__rfm_check__.short_name}.log'
+        )
         self.stream = self._streams.get(self.baseFilename, None)
         super().emit(record)
         self._streams[self.baseFilename] = self.stream
diff --git a/reframe/core/meta.py b/reframe/core/meta.py
index 3abae0df..f3e151b8 100644
--- a/reframe/core/meta.py
+++ b/reframe/core/meta.py
@@ -423,6 +423,8 @@ class RegressionTestMeta(type):
             obj._rfm_unique_name = fixt_name
             obj._rfm_fixt_data = fixt_data
             obj._rfm_is_fixture = True
+        else:
+            obj._rfm_unique_name = cls.variant_name(variant_num)
 
         # Set the variables passed to the constructor
         for k, v in fixt_vars.items():
diff --git a/reframe/core/pipeline.py b/reframe/core/pipeline.py
index f189ef3c..4911cff8 100644
--- a/reframe/core/pipeline.py
+++ b/reframe/core/pipeline.py
@@ -14,6 +14,7 @@ __all__ = [
 
 
 import glob
+import hashlib
 import inspect
 import itertools
 import numbers
@@ -281,8 +282,11 @@ class RegressionTest(RegressionMixin, jsonext.JSONSerializable):
     #: A detailed description of the test.
     #:
     #: :type: :class:`str`
-    #: :default: ``self.display_name``
-    descr = variable(str, loggable=True)
+    #: :default: ``''``
+    #:
+    #: .. versionchanged:: 4.0
+    #:    The default value is now the empty string.
+    descr = variable(str, value='', loggable=True)
 
     #: The path to the source file or source directory of the test.
     #:
@@ -959,13 +963,6 @@ class RegressionTest(RegressionMixin, jsonext.JSONSerializable):
 
     @deferrable
     def __rfm_init__(self, prefix=None):
-        if not self.is_fixture() and not hasattr(self, '_rfm_unique_name'):
-            self._rfm_unique_name = type(self).variant_name(self.variant_num)
-
-        # Pass if descr is a required variable.
-        if not hasattr(self, 'descr'):
-            self.descr = self.display_name
-
         self._perfvalues = {}
 
         # Static directories of the regression check
@@ -1089,9 +1086,9 @@ class RegressionTest(RegressionMixin, jsonext.JSONSerializable):
     def name(self):
         '''The name of the test.
 
-        This is an alias of :attr:`unique_name`.
+        This is an alias of :attr:`display_name`.
         '''
-        return self.unique_name
+        return self.display_name
 
     @loggable
     @property
@@ -1146,6 +1143,42 @@ class RegressionTest(RegressionMixin, jsonext.JSONSerializable):
 
         return self._rfm_display_name
 
+    @loggable
+    @property
+    def hashcode(self):
+        if hasattr(self, '_rfm_hashcode'):
+            return self._rfm_hashcode
+
+        m = hashlib.sha256()
+        if self.is_fixture:
+            m.update(self.unique_name.encode('utf-8'))
+        else:
+            basename, *params = self.display_name.split(' %')
+            m.update(basename.encode('utf-8'))
+            for p in sorted(params):
+                m.update(p.encode('utf-8'))
+
+        self._rfm_hashcode = m.hexdigest()[:8]
+        return self._rfm_hashcode
+
+    @loggable
+    @property
+    def short_name(self):
+        '''A short version of the test's display name.
+
+        The shortened version coincides with the :attr:`unique_name` for
+        simple tests and combines the test's class name and a hash code for
+        parameterised tests.
+
+        .. versionadded:: 4.0.0
+
+        '''
+
+        if self.unique_name != self.display_name:
+            return f'{type(self).__name__}_{self.hashcode}'
+        else:
+            return self.unique_name
+
     @property
     def current_environ(self):
         '''The programming environment that the regression test is currently
@@ -1397,7 +1430,7 @@ class RegressionTest(RegressionMixin, jsonext.JSONSerializable):
            method may be called at any point of the test's lifetime.
         '''
 
-        ret = self.display_name
+        ret = f'{self.display_name} /{self.hashcode}'
         if self.current_partition:
             ret += f' @{self.current_partition.fullname}'
 
@@ -1508,11 +1541,11 @@ class RegressionTest(RegressionMixin, jsonext.JSONSerializable):
             runtime = rt.runtime()
             self._stagedir = runtime.make_stagedir(
                 self.current_system.name, self._current_partition.name,
-                self._current_environ.name, self.unique_name
+                self._current_environ.name, self.short_name
             )
             self._outputdir = runtime.make_outputdir(
                 self.current_system.name, self._current_partition.name,
-                self._current_environ.name, self.unique_name
+                self._current_environ.name, self.short_name
             )
         except OSError as e:
             raise PipelineError('failed to set up paths') from e
@@ -1540,13 +1573,12 @@ class RegressionTest(RegressionMixin, jsonext.JSONSerializable):
                           **job_opts)
 
     def _setup_build_job(self, **job_opts):
-        self._build_job = self._create_job(f'rfm_{self.unique_name}_build',
+        self._build_job = self._create_job(f'rfm_build',
                                            self.local or self.build_locally,
                                            **job_opts)
 
     def _setup_run_job(self, **job_opts):
-        self._job = self._create_job(f'rfm_{self.unique_name}_job',
-                                     self.local, **job_opts)
+        self._job = self._create_job(f'rfm_job', self.local, **job_opts)
 
     def _setup_perf_logging(self):
         self._perf_logger = logging.getperflogger(self)
diff --git a/reframe/frontend/cli.py b/reframe/frontend/cli.py
index 0412fd61..b4754532 100644
--- a/reframe/frontend/cli.py
+++ b/reframe/frontend/cli.py
@@ -73,6 +73,7 @@ def list_checks(testcases, printer, detailed=False, concretized=False):
                 unique_checks.add(v.check.unique_name)
 
         if depth:
+            name_info = f'{u.check.display_name} /{u.check.hashcode}'
             tc_info = ''
             details = ''
             if concretized:
@@ -80,17 +81,16 @@ def list_checks(testcases, printer, detailed=False, concretized=False):
 
             location = inspect.getfile(type(u.check))
             if detailed:
-                details = f' [id: {u.check.unique_name}, file: {location!r}]'
+                details = f' [variant: {u.check.variant_num}, file: {location!r}]'
 
-            lines.append(
-                f'{prefix}^{u.check.display_name}{tc_info}{details}'
-            )
+            lines.append(f'{prefix}^{name_info}{tc_info}{details}')
 
         return lines
 
     # We need the leaf test cases to be printed at the leftmost
     leaf_testcases = list(t for t in testcases if t.in_degree == 0)
     for t in leaf_testcases:
+        name_info = f'{t.check.display_name} /{t.check.hashcode}'
         tc_info = ''
         details = ''
         if concretized:
@@ -98,12 +98,12 @@ def list_checks(testcases, printer, detailed=False, concretized=False):
 
         location = inspect.getfile(type(t.check))
         if detailed:
-            details = f' [id: {t.check.unique_name}, file: {location!r}]'
+            details = f' [variant: {t.check.variant_num}, file: {location!r}]'
 
         # if not concretized and t.check.name not in unique_checks:
         if concretized or (not concretized and
                            t.check.unique_name not in unique_checks):
-            printer.info(f'- {t.check.display_name}{tc_info}{details}')
+            printer.info(f'- {name_info}{tc_info}{details}')
 
         if not t.check.is_fixture():
             unique_checks.add(t.check.unique_name)
diff --git a/reframe/frontend/executors/__init__.py b/reframe/frontend/executors/__init__.py
index 267c95c8..b06a1abb 100644
--- a/reframe/frontend/executors/__init__.py
+++ b/reframe/frontend/executors/__init__.py
@@ -416,9 +416,10 @@ class RegressionTask:
     def info(self):
         '''Return an info string about this task.'''
         name = self.check.display_name
+        hashcode = self.check.hashcode
         part = self.testcase.partition.fullname
         env  = self.testcase.environ.name
-        return f'{name} @{part}+{env}'
+        return f'{name} /{hashcode} @{part}+{env}'
 
 
 class TaskEventListener(abc.ABC):
diff --git a/reframe/frontend/filters.py b/reframe/frontend/filters.py
index efdb95c9..20441a18 100644
--- a/reframe/frontend/filters.py
+++ b/reframe/frontend/filters.py
@@ -41,13 +41,16 @@ def have_not_name(patt):
 
 def have_any_name(names):
     rt = runtime()
-    exact_matches = []
+    variant_matches = []
+    hash_matches = []
     regex_matches = []
     for n in names:
         if '@' in n:
             test, _, variant = n.rpartition('@')
             if variant.isdigit():
-                exact_matches.append((test, int(variant)))
+                variant_matches.append((test, int(variant)))
+        elif n.startswith('/'):
+            hash_matches.append(n[1:])
         else:
             regex_matches.append(n)
 
@@ -57,12 +60,17 @@ def have_any_name(names):
         regex = None
 
     def _fn(case):
-        # Check if we have an exact match
-        for m in exact_matches:
+        # Check the variant matches
+        for m in variant_matches:
             cls_name = type(case.check).__name__
             if (cls_name, case.check.variant_num) == m:
                 return True
 
+        # Check hash matches
+        for m in hash_matches:
+            if m == case.check.hashcode:
+                return True
+
         display_name = case.check.display_name.replace(' ', '')
         if regex:
             return regex.match(display_name)


Test patch:
diff --git a/unittests/test_dependencies.py b/unittests/test_dependencies.py
index e7378220..dec8ef56 100644
--- a/unittests/test_dependencies.py
+++ b/unittests/test_dependencies.py
@@ -37,7 +37,7 @@ class Node:
                     self.ename == other.ename)
 
         if isinstance(other, executors.TestCase):
-            return (self.cname == other.check.name and
+            return (self.cname == other.check.unique_name and
                     self.pname == other.partition.fullname and
                     self.ename == other.environ.name)
 
@@ -450,9 +450,9 @@ def test_build_deps(loader, default_exec_ctx):
     check_e0._current_environ = Environment('e0')
     check_e1._current_environ = Environment('e1')
 
-    assert check_e0.getdep('Test0', 'e0', 'p0').name == 'Test0'
-    assert check_e0.getdep('Test0', 'e1', 'p0').name == 'Test0'
-    assert check_e1.getdep('Test0', 'e1', 'p0').name == 'Test0'
+    assert check_e0.getdep('Test0', 'e0', 'p0').unique_name == 'Test0'
+    assert check_e0.getdep('Test0', 'e1', 'p0').unique_name == 'Test0'
+    assert check_e1.getdep('Test0', 'e1', 'p0').unique_name == 'Test0'
     with pytest.raises(DependencyError):
         check_e0.getdep('TestX_deprecated', 'e0', 'p0')
 
@@ -605,7 +605,7 @@ def test_skip_unresolved_deps(make_exec_ctx):
     )
     assert len(skipped_cases) == 6
 
-    skipped_tests = {c.check.name for c in skipped_cases}
+    skipped_tests = {c.check.unique_name for c in skipped_cases}
     assert skipped_tests == {'t1', 't2', 't3'}
 
 
@@ -615,9 +615,9 @@ def assert_topological_order(cases, graph):
     tests = util.OrderedSet()
     for c in cases:
         check, part, env = c
-        cases_order.append((check.name, part.fullname, env.name))
-        tests.add(check.name)
-        visited_tests.add(check.name)
+        cases_order.append((check.unique_name, part.fullname, env.name))
+        tests.add(check.unique_name)
+        visited_tests.add(check.unique_name)
 
         # Assert that all dependencies of c have been visited before
         for d in graph[c]:
@@ -625,7 +625,7 @@ def assert_topological_order(cases, graph):
                 # dependency points outside the subgraph
                 continue
 
-            assert d.check.name in visited_tests
+            assert d.check.unique_name in visited_tests
 
     # Check the order of systems and prog. environments
     # We are checking against all possible orderings
@@ -739,7 +739,7 @@ def test_toposort(default_exec_ctx):
     cases_by_level = {}
     for c in cases:
         cases_by_level.setdefault(c.level, set())
-        cases_by_level[c.level].add(c.check.name)
+        cases_by_level[c.level].add(c.check.unique_name)
 
     assert cases_by_level[0] == {'t0', 't5'}
     assert cases_by_level[1] == {'t1', 't6', 't7'}
@@ -784,7 +784,7 @@ def test_toposort_subgraph(default_exec_ctx):
     cases_by_level = {}
     for c in cases:
         cases_by_level.setdefault(c.level, set())
-        cases_by_level[c.level].add(c.check.name)
+        cases_by_level[c.level].add(c.check.unique_name)
 
     assert cases_by_level[1] == {'t3'}
     assert cases_by_level[2] == {'t4'}
diff --git a/unittests/test_filters.py b/unittests/test_filters.py
index e052061e..3ebbbd40 100644
--- a/unittests/test_filters.py
+++ b/unittests/test_filters.py
@@ -65,6 +65,8 @@ def test_have_any_name(sample_cases):
                              sample_cases)
     assert 2 == count_checks(filters.have_any_name(['(?i)check1|CHECK2']),
                              sample_cases)
+    assert 1 == count_checks(filters.have_any_name(['/e2ae5cc6']),
+                             sample_cases)
 
 
 def test_have_any_name_param_test(sample_param_cases):
@@ -81,6 +83,8 @@ def test_have_any_name_param_test(sample_param_cases):
                              sample_param_cases)
     assert 0 == count_checks(filters.have_any_name(['_X@12']),
                              sample_param_cases)
+    assert 2 == count_checks(filters.have_any_name(['/023313dc', '/efddbc6c']),
+                             sample_param_cases)
     assert 2 == count_checks(filters.have_any_name(['_X@0', '_X@1']),
                              sample_param_cases)
     assert 12 == count_checks(filters.have_any_name(['_X@0', '_X.*']),
diff --git a/unittests/test_fixtures.py b/unittests/test_fixtures.py
index 6563fb7a..f9160a5a 100644
--- a/unittests/test_fixtures.py
+++ b/unittests/test_fixtures.py
@@ -531,7 +531,9 @@ def test_overlapping_registries(ctx_sys, simple_test,
     # correctly.
     assert len(inst) == 1
     assert inst[0].v == 2
-    assert inst[0].name == list(diff_reg[simple_fixture().cls].keys())[0]
+    assert inst[0].unique_name == list(
+        diff_reg[simple_fixture().cls].keys()
+    )[0]
     assert inst[0].valid_systems == ['sys1:p0']
     assert inst[0].valid_prog_environs == ['e0']
 
diff --git a/unittests/test_logging.py b/unittests/test_logging.py
index 551b6b00..60acf61a 100644
--- a/unittests/test_logging.py
+++ b/unittests/test_logging.py
@@ -467,7 +467,7 @@ def test_logging_context_check(default_exec_ctx, logfile, fake_check):
 
     rlog.getlogger().error('error outside context')
     assert _found_in_logfile(
-        f'_FakeCheck_1: {sys.argv[0]}: error from context', logfile
+        f'_FakeCheck %param=10: {sys.argv[0]}: error from context', logfile
     )
     assert _found_in_logfile(
         f'reframe: {sys.argv[0]}: error outside context', logfile