Bug: C-C mochitest does not run if we have some symlinks in related directories. (|make mozmill| did work with the symlinks.)
Status: NEW
Priority: --
Created: 2020-01-10 18:24:39+00:00
Resolved: 2022-10-11 23:49:51+00:00

(This is a quote of a discussion from Maildev mailing list.)
mochitest runtest.py has a too strict path consistency check or it is not handling
symlinks in an intelligent manner. |make mozmill| did not have any such issues with symlinks in my filesystem.

-- begin quote ---
On 2019/12/25 20:15, Magnus Melin wrote:
> ./mach test comm/mail/test/browser/composition/browser_imageInsertionDialog.js 

Thanks.

BTW, This is under Linux amd64.

I updated the source just in case this morning. I think several files were updated under ./comm subdirecotry as well as many under ./mozilla main directory.

Rebuilt TB.

Unfortunately, it is still NG, and I found the cause, I think.

The tail of the log shows: (there are a few locally added error logs.)
```
++ echo 'Your current directory is'
Your current directory is
++ pwd
/NREF-COMM-CENTRAL/mozilla
++ /NREF-COMM-CENTRAL/mozilla/mach --log-no-times test comm/mail/test/browser/composition/browser_imageInsertionDialog.js
kind=None
environ.get('TERM', 'unknown')=xterm-256color
environ.get('TERM', 'linux')=xterm-256color
kind=None
environ.get('TERM', 'unknown')=xterm-256color
environ.get('TERM', 'linux')=xterm-256color
 0:01.07 INFO Checking for ssltunnel processes...
 0:01.09 INFO Checking for xpcshell processes...
Error running mach:

    ['--log-no-times', 'test', 'comm/mail/test/browser/composition/browser_imageInsertionDialog.js']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

AssertionError

  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mach_commands.py", line 387, in test
    argv=extra_args, test_objects=tests, **kwargs)
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mach/mach/registrar.py", line 152, in dispatch
    return self._run_command_handler(handler, context=context, **kwargs)
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mach/mach/registrar.py", line 109, in _run_command_handler
    result = fn(**kwargs)
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mochitest/mach_commands.py", line 447, in run_mochitest_general
    **harness_args)
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mochitest/mach_commands.py", line 142, in run_desktop_test
    result = mochitest.run_test_harness(parser, options)
  File "/KERNEL-SRC/moz-obj-dir/objdir-tb3/_tests/testing/mochitest/runtests.py", line 3146, in run_test_harness
    result = runner.runTests(options)
  File "/KERNEL-SRC/moz-obj-dir/objdir-tb3/_tests/testing/mochitest/runtests.py", line 2601, in runTests
    tests = self.getActiveTests(options)
  File "/KERNEL-SRC/moz-obj-dir/objdir-tb3/_tests/testing/mochitest/runtests.py", line 1494, in getActiveTests
    assert pathAbs.startswith(self.testRootAbs)

real    0m4.087s
user    0m3.472s
sys    0m0.618s
```

Just to be sure,
I did clobber,
re-configure,
rebuilt.
Still no go.

Looks to me there is something wrong with the assert().

I inserted a couple of log.warnings statements in runtests.py just before the problematic assert().
```
            pathAbs = os.path.abspath(test['path'])
            self.log.warning('INFO: pathAbs=%s' % pathAbs)
            self.log.warning('INFO: self.testRootAbs=%s' % self.testRootAbs)
            assert pathAbs.startswith(self.testRootAbs)
            tp = pathAbs[len(self.testRootAbs):].replace('\\', '/').strip('/')
```
Aha, I think there *ARE* problems.

From the new log:
```
 0:00.49 WARNING INFO: pathAbs=/KERNEL-SRC/moz-obj-dir/objdir-tb3/_tests/testing/mochitest/browser/comm/mail/test/browser/composition/browser_imageInsertionDialog.js
 0:00.49 WARNING INFO: self.testRootAbs=/NEW-SSD/moz-obj-dir/objdir-tb3/_tests/testing/mochitest/browser
Error running mach:
```

Obviously, the pathAbs DOES NOT start with self.testRootAbs textually.

However, my /NREF-COMM-CENTRAL/mozilla main directory and an intermediate path to my MOZ_OBJ are actually symlinks to
the final directories.  And pathAbs DOES start with self.testRootAbs if symlinks are resolved.
(CORRECTION: To be exact, only the /KERNEL-SRC/moz-obj-dir part of the symlink is resolved.
We all know that the test files under MOZ_OBJ is actually symlinks to the ORIGINAL source files. So if we resolve THE final symlink in the path, we end up in the original source files and they DO NOT start with self.testRootAbs at all.)

Can this be a problem?

ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$ ls -l /NREF-COMM-CENTRAL
lrwxrwxrwx 1 root root 26 Oct 10 17:31 /NREF-COMM-CENTRAL -> /NEW-SSD/NREF-COMM-CENTRAL/

from my mozconfig:
mk_add_options MOZ_OBJDIR=/KERNEL-SRC/moz-obj-dir/objdir-tb3

ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$ ls -l /KERNEL-SRC/moz-obj-dir
lrwxrwxrwx 1 root root 20 Oct  9 21:00 /KERNEL-SRC/moz-obj-dir -> /NEW-SSD/moz-obj-dir/
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$

Obnoxious thing is that |make mozmill| used to work without any issues with these symlink acrobatics at all.
(Symlink acrobatics were used to move TB source and object directories to new disks in the past without the need to update various shell scripts to test and analyze the log.)

I am tempted to remove the problematic assert for now and see how far I can go.

Should I file a bugzilla? I bet I should do. I will report whether removing the assert will get me far.
--- end quote ---

Thus I am reporting this.

By commenting out the assert(), the test seems to be invoked finally, but I see another assert which prevents the test to work locally as follows. I will file another bug on that.


```
 0:02.04 INFO Application command: /NEW-SSD/moz-obj-dir/objdir-tb3/dist/bin/thunderbird -marionette -foreground -profile /COMM-CENTRAL/TMP-DIR/tmpIWM7ka.mozrunner
 0:02.07 INFO runtests.py | Application pid: 119773
 0:02.07 Started process `GECKO(119773)`
 0:02.12 GECKO(119773) [119773, Unnamed thread 556442d30950] WARNING: XPCOM objects created/destroyed from static ctor/dtor: file /NEW-SSD/NREF-COMM-CENTRAL/mozilla/xpcom/base/nsTraceRefcnt.cpp, line 198
 0:02.12 GECKO(119773) ### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to /COMM-CENTRAL/TMP-DIR/tmpIWM7ka.mozrunner/runtests_leaks.log
 0:02.12 GECKO(119773) Assertion failure: !mRawPtr, at /NEW-SSD/moz-obj-dir/objdir-tb3/dist/include/mozilla/StaticPtr.h:51
 0:02.45 GECKO(119773) #01: __static_initialization_and_destruction_0(int, int) (/NEW-SSD/moz-obj-dir/objdir-tb3/dist/include/mozilla/Assertions.h:42)
 0:02.46 GECKO(119773) #02: _GLOBAL__sub_I_Unified_cpp_xpcom_base2.cpp (/NEW-SSD/NREF-COMM-CENTRAL/mozilla/xpcom/base/nsWeakReference.cpp:160)
 0:02.53 GECKO(119773) #03: call_init (/build/glibc-suXNNi/glibc-2.29/elf/dl-init.c:71)
 0:02.53 GECKO(119773) #04: _dl_init (/build/glibc-suXNNi/glibc-2.29/elf/dl-init.c:118)
 0:02.53 GECKO(119773) #05: dl_open_worker (/build/glibc-suXNNi/glibc-2.29/elf/dl-open.c:520)
 0:02.67 GECKO(119773) #06: __GI__dl_catch_exception (/build/glibc-suXNNi/glibc-2.29/elf/dl-error-skeleton.c:197)
 0:02.67 GECKO(119773) #07: _dl_open (/build/glibc-suXNNi/glibc-2.29/elf/dl-open.c:599)
 0:02.73 GECKO(119773) #08: dlopen_doit (/build/glibc-suXNNi/glibc-2.29/dlfcn/dlopen.c:66 (discriminator 8))
 0:02.73 GECKO(119773) #09: __GI__dl_catch_exception (/build/glibc-suXNNi/glibc-2.29/elf/dl-error-skeleton.c:197)
 0:02.74 GECKO(119773) #10: __GI__dl_catch_error (/build/glibc-suXNNi/glibc-2.29/elf/dl-error-skeleton.c:216)
 0:02.74 GECKO(119773) #11: _dlerror_run (/build/glibc-suXNNi/glibc-2.29/dlfcn/dlerror.c:174)
 0:02.74 GECKO(119773) #12: __dlopen (/build/glibc-suXNNi/glibc-2.29/dlfcn/dlopen.c:87)
 0:02.82 GECKO(119773) #13: GetLibHandle(char const*) (/NEW-SSD/NREF-COMM-CENTRAL/mozilla/xpcom/glue/standalone/nsXPCOMGlue.cpp:85)
 0:02.82 GECKO(119773) #14: ReadDependentCB(char const*, mozilla::LibLoadingStrategy) (/NEW-SSD/NREF-COMM-CENTRAL/mozilla/xpcom/glue/standalone/nsXPCOMGlue.cpp:135)
 0:02.82 GECKO(119773) #15: XPCOMGlueLoad(char const*, mozilla::LibLoadingStrategy) (/NEW-SSD/NREF-COMM-CENTRAL/mozilla/xpcom/glue/standalone/nsXPCOMGlue.cpp:309)
 0:02.82 GECKO(119773) #16: mozilla::GetBootstrap(char const*, mozilla::LibLoadingStrategy) (/NEW-SSD/NREF-COMM-CENTRAL/mozilla/xpcom/glue/standalone/nsXPCOMGlue.cpp:377)
 0:02.82 GECKO(119773) #17: InitXPCOMGlue() (/NEW-SSD/moz-obj-dir/objdir-tb3/dist/include/mozilla/UniquePtr.h:314)
 0:02.82 GECKO(119773) #18: main (/NEW-SSD/NREF-COMM-CENTRAL/mozilla/comm/mail/app/nsMailApp.cpp:314 (discriminator 4))
 0:02.82 GECKO(119773) #19: __libc_start_main (/build/glibc-suXNNi/glibc-2.29/csu/../csu/libc-start.c:342)
 0:02.82 GECKO(119773) #20: _start (/NEW-SSD/moz-obj-dir/objdir-tb3/dist/bin/thunderbird)
 0:02.82 GECKO(119773) #21: ??? (???:???)
 2:02.12 INFO runtests.py | Waiting for browser...
 2:02.12 Main app process: killed by SIGSEGV
 2:02.12 INFO Buffered messages finished
 2:02.12 ERROR TEST-UNEXPECTED-FAIL | automation.py | application terminated with exit code -11
 2:02.12 INFO runtests.py | Application ran for: 0:02:00.080468
 2:02.12 INFO zombiecheck | Reading PID log: /COMM-CENTRAL/TMP-DIR/tmpS4XB8hpidlog
Traceback (most recent call last):
  File "/NEW-SSD/moz-obj-dir/objdir-tb3/_tests/testing/mochitest/runtests.py", line 2863, in doTests
    e10s=options.e10s
  File "/NEW-SSD/moz-obj-dir/objdir-tb3/_tests/testing/mochitest/runtests.py", line 2382, in runApp
    raise exc(value).with_traceback(tb)
AttributeError: 'timeout' object has no attribute 'with_traceback'
 2:02.13 ERROR Automation Error: Received unexpected exception while running application

```