Repository: spack/spack

Issue:
openfoam: modified 'setup_run_environment'
Added the setting of environment variables at running openfoam when using the FJ compiler.
(I confirmed that Ver.1812 can be run.)


Discussion:
> Added the setting of environment variables at running openfoam when using the FJ compiler.
> (I confirmed that Ver.1812 can be run.)

Does this mean the change should be tied to that version?  Or does it apply in general?
It is apply in general.
However, I have confirmed the execution only in Ver.1812.
@olesenm As the maintainer, can you please take a look at this PR?
In the default settings (if you does not set the FOAM_SIGFPE variable), OpenFOAM catches floating-point exceptions.
This is the same behavior as if the FOAM_SIGFPE variable is true.
The process of catching floating-point exceptions in OpenFOAM uses the feenableexcept included in the GNU C library. 
If FOAM_SIGFPE variable is enabled and the Fujitsu compiler is used, the program will result in a run-time error even if there is no problem with the program.
Therefore, if you run OpenFOAM using the Fujitsu compiler, you must set FOAM_SIGFPE to false.
> In the default settings (if you does not set the FOAM_SIGFPE variable), OpenFOAM catches floating-point exceptions.
> This is the same behavior as if the FOAM_SIGFPE variable is true.
> The process of catching floating-point exceptions in OpenFOAM uses the feenableexcept included in the GNU C library.
> If FOAM_SIGFPE variable is enabled and the Fujitsu compiler is used, the program will result in a run-time error even if there is no problem with the program.
> Therefore, if you run OpenFOAM using the Fujitsu compiler, you must set FOAM_SIGFPE to false.

I still don't get this. If we look at the [sigFpe.C code](https://develop.openfoam.com/Development/openfoam/-/blob/master/src/OSspecific/POSIX/signals/sigFpe.C), the environment handling is wrapped by the `isTrue` local function (line 68).
If the env variable is unset, then it falls through to the default value. This all wrapped by the `requested()` method (line 177).

Defining it as _false_ will simply ensure that the `switchFpe_` value is not taken. It does not do any other setting/unsetting on its own.

So the question to ask is how/why is `switchFpe_` getting set?

The compile-time default is set in line 57 and is false, but you see that it has the optimisationSwitch "trapFpe".

This traces back to the [etc/controlDict entry](https://develop.openfoam.com/Development/openfoam/-/blob/master/etc/controlDict#L130).

This is the thing that I guess you want to reconfigure for Fujitsu. I still think that using an env variable is too fragile.

@ShunIto-FJ  I have opened an issue
https://develop.openfoam.com/Development/openfoam/-/issues/1998

@ShunIto-FJ - any followup?
If we _really_ wanted to make the change for spack, then filtering/editing the etc/controlDict would be the way to go.
However, in my experience the SIGFPE issues are often related to a programming error, either within OpenFOAM itself or in user coding. In some older code the redistributePar may have exposed uninitialized memory that provoke problems. Unfortunately can't remember the exact details.  It would help if you could post when/where the issues arise.
@olesenm The reason for setting the runtime environment variables is that a runtime error occurred when using the Fujitsu compiler (clang mode) in the aarch64 environment.
We are currently checking the detailed cause of the error with the Fujitsu compiler developer.
Could you wait for Fujitsu compiler developer answer?
> @olesenm The reason for setting the runtime environment variables is that a runtime error occurred when using the Fujitsu compiler (clang mode) in the aarch64 environment.
> We are currently checking the detailed cause of the error with the Fujitsu compiler developer.
> Could you wait for Fujitsu compiler developer answer?

Perfect - can also tag me on https://develop.openfoam.com/Development/openfoam/-/issues if you want
@ShunIto-FJ @olesenm Hi, I'm a developer of Fujitsu compiler (clang mode). I'll describe the problem.

Clang/LLVM has a problem that it performs optimizations with the assumption that floating-point operations have no side effects. It means that Clang/LLVM may generate code which causes unneeded SIGFPE. This problem is [known in the LLVM community](https://bugs.llvm.org/show_bug.cgi?id=6050). One of the simplest cases is in [PR24343](https://bugs.llvm.org/show_bug.cgi?id=24343#c0). It says that the following code causes SIGFPE (divide-by-zero).

```c
#include <fenv.h>

__attribute__((__noinline__)) double foo(int n, double d)
{
  if (n > 0)
    d = 1.0 / d;
  return d;
}

int main(void)
{
  feenableexcept(FE_DIVBYZERO);
  return (int)foo(0, 0.0);
}
```

Problematic code is generated by processor-dependent and/or processor-independent optimizations of LLVM. It means that whether SIGFPE actually occurs or not depends on the source code, the target processor, and the optimization level.

Fujitsu compiler clang mode is based on Clang/LLVM and has potentially the same problem. In our case, our custom loop vectorizer generates code which may cause divide-by-zero in a tail loop which has a division operation.

This is the reason why we need `FOAM_SIGFPE=false`. As I wrote above, I think Clang/LLVM also potentially needs `FOAM_SIGFPE=false`.

FYI: Recent Clang/LLVM [introduced `-ftrapping-math` and `-ffp-exception-behavior=...` options](https://github.com/llvm/llvm-project/commit/7f9b5138470db1dc58f3bc05631284c653c9ed7a). Specifying one of these option for recent Clang/LLVM may get rid of the problem but I'm not sure these options are implemented perfectly.

@kawashima-fj Thank you for following me with a detailed explanation.

@olesenm I agree with controlling the functionality of OpenFOAM's floating point exception traps with trapFpe in etc/controlDict instead of environment variables.
Should I close this pull request?
OK to close after you have tested and verified that my pull request correctly solves your original problem. If that is indeed the case, then wait a tiny bit until _that_ pull request also gets merged (we don't want to make any unnecessary accidents) 

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/openfoam/package.py b/var/spack/repos/builtin/packages/openfoam/package.py
--- a/var/spack/repos/builtin/packages/openfoam/package.py
+++ b/var/spack/repos/builtin/packages/openfoam/package.py
@@ -405,10 +405,7 @@ def setup_minimal_environment(self, env):
     def setup_build_environment(self, env):
         """Sets the build environment (prior to unpacking the sources).
         """
-        # Avoid the exception that occurs at runtime
-        # when building with the Fujitsu compiler.
-        if self.spec.satisfies('%fj'):
-            env.set('FOAM_SIGFPE', 'false')
+        pass
 
     def setup_run_environment(self, env):
         """Sets the run environment (post-installation).
@@ -549,6 +546,16 @@ def patch(self):
                     rcfile,
                     backup=False)
 
+    def configure_trapFpe_off(self):
+        """Disable trapFpe handling.
+        Seems to be needed for several clang-derivatives.
+        """
+        # Set 'trapFpe 0' in etc/controlDict
+        controlDict = 'etc/controlDict'
+        if os.path.exists(controlDict):
+            filter_file(r'trapFpe\s+\d+\s*;', 'trapFpe 0;',
+                        controlDict, backup=False)
+
     @when('@1812: %fj')
     @run_before('configure')
     def make_fujitsu_rules(self):
@@ -560,6 +567,7 @@ def make_fujitsu_rules(self):
         arch_rules = 'wmake/rules/linuxARM64'  # self.arch
         src = arch_rules + 'Clang'
         dst = arch_rules + 'Fujitsu'  # self.compiler
+        self.configure_trapFpe_off()  # LLVM may falsely trigger FPE
 
         if os.path.exists(dst):
             return


Test patch: