Bug: Two cases of heap-use-after-free during local mochitest under linux using ASAN build
Status: NEW
Priority: --
Created: 2024-11-07 01:29:43+00:00
Resolved: 2024-11-07 10:04:49+00:00

I have detected two cases of heap-use-after-free during the mochitest of C-C TB using an ASAN build.
This is tested under Debian GNU/Linux.
M-C/C-C tree was updated the day before.

One of the  heap-use-after-free possibly is one of the mysterious reasons of crash at shutdown time.

The tests of mochitest suite that triggered the heap-use-after-free 
are
```
- comm/mail/base/test/browser/browser_getMessages.js
- comm/mail/test/browser/openpgp/composition/browser_composeSigned2.js
```

But given the asynchronous nature, the bug may not manifest itself when we repeat the tests.

The output of the following command is as follows where log1766-mochitest.txt is the log record of running
mochitest locally.
```
egrep -a "(TEST[-_]START|TEST[-_]END|TEST[-_]OK|heap-use-after-free)" log1766-mochitest.txt  | gawk -f ~/Dropbox/TB-DIR/ignore-immediate-test-start-and-end-pairs.awk
```
The awk script picks up only the TEST_START and TEST_END lines of tests where the "heap-use-after-free" string is found.

```
00:09.86 TEST_END: Test OK. Subtests passed 0/1. Unexpected 1
28:16.64 TEST_START: comm/mail/base/test/browser/browser_getMessages.js
29:52.17 GECKO(2125195) ==2125195==ERROR: AddressSanitizer: heap-use-after-free on address 0x53100029a190 at pc 0x7fb23536d7c5 bp 0x7ffd34021060 sp 0x7ffd34021058
29:53.52 GECKO(2125195) SUMMARY: AddressSanitizer: heap-use-after-free /NEW-SSD/NREF-COMM-CENTRAL/mozilla/comm/mailnews/db/mork/morkRow.h:184:23 in morkRow::EqualRow(morkRow const*) const
30:00.71 TEST_END: Test OK. Subtests passed 84/84. Unexpected 0
149:17.78 TEST_START: comm/mail/test/browser/openpgp/composition/browser_composeSigned2.js
149:29.64 GECKO(2168524) ==2168524==ERROR: AddressSanitizer: heap-use-after-free on address 0x5080001200a0 at pc 0x7f3ddb45cc4a bp 0x7ffca0ed2360 sp 0x7ffca0ed2358
149:30.80 GECKO(2168524) SUMMARY: AddressSanitizer: heap-use-after-free /NEW-SSD/ASAN-OBJ-DIR/objdir-tb3/dist/include/mozilla/RefPtr.h:49:40 in mozilla::RefPtrTraits<nsISupports>::Release(nsISupports*)
149:42.51 TEST_END: Test OK. Subtests passed 5/5. Unexpected 0
```

Please note that the test has passed for the 1st case, and the heap-use-after-free occurs only during shutdown phase. So the test pass, but we may crash due to the invalid memory access.
The second case is indeed a crash case during normal processing.

The crash may happen depending on the particular library and/or other factors such as the memory allocation history, ordering of asynchronous thread execution, etc.

I am attaching the stacktrace from the two instances of heap-use-after-free printed by the ASAN runtime.
Note that, for local debugging to figure out why valgrind fails, I modified subcommand that is invoked by |mach| to print some internal information and that is at the end of each stack dump.

Unlike Bug 1677202, this bug is NOT something specific to the DEBUG build, but
is related to a generic  processing that invokes GC (garbage collection) and that is when this heap-use-after-free occurs.
It seems that some pointers are freed by GC, but they still remain in the application data structure, and
the destruction of such data structure caused an automatic destruction of certain objects which lead to
the access via the freed pointer.
That is my brief guess after looking at the stacktraces.

I am not even sure if  TB is robust in the face of GC during normal time even, and the second crash happens during normal processing, not at shutdown time.

I have not seen this type of error in the last six months or so, unrelated to DEBUG code, since I began testing using ASAN build. So there may have been a recent change in the code involved in the stacktrace, which may have caused the error.

I think we need to introduce some type of ordering of the destruction of objects based on their types during GC.
This may seem unnatural to, say, pure Lisp programmers, but the convoluted data structure of C-C TB requires such an approach to become free of the heap-use-after-free problems at shutdown IMHO.

NOTE: After writing all this initially, I now realize that the second heap-use-after-free seems to have caused the test to abort. I don't see the proper TEST_END line for test comm/mail/test/browser/openpgp/composition/browser_composeSigned2.js
So it may crash C-C TB during normal abort time.
The awk script was not robust enough to detect this situation where TEST_END is not found due to premature abort of the executable. The above short listing picked up the TEST_END line for the NEXT test that followed 
comm/mail/test/browser/openpgp/composition/browser_composeSigned2.js, namely,
comm/mail/test/browser/pref-window/browser_fontChooser.js

EDIT: I edited the post to make it clear that the second error is not at shutodown time whereas the first one happened during shutdown processing.