Repository: pantsbuild/pants

Issue:
Panicked while processing panic
It seems like somewhere in the 2.2.x series, a panic-while-panicking was introduced in an unknown location. We should track it down before 2.2.0 final.

```
09:59:57.70 [INFO] Resolving new plugins...:
  toolchain.pants.plugin==0.1.0
10:00:24.15 [WARN] /home/travis/.pex/installed_wheels/746f296551eae19b681843508031a35999059f46/pex-2.1.24-py2.py3-none-any.whl/pex/vendor/_vendored/packaging/packaging/version.py:130: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  DeprecationWarning,

10:00:26.15 [WARN] Unmatched glob from src/python/pants/engine/internals:native_engine's `sources` field: "src/python/pants/engine/internals/native_engine.so.metadata"
thread panicked while processing panic. aborting.
```
(from https://travis-ci.com/github/pantsbuild/pants/jobs/463835089)


Discussion:
Potentially related? https://github.com/dgrunwald/rust-cpython/pull/244 ... We're doing a lot more cancellation than we were.
Updated all Rust dependencies (including `cpython`) in #11366. Will hold off on this one to see whether that had any impact.
We haven't had any repros of this after #11366. Calling it fixed.
This might still be around. I'll spend some time today trying to repro it to get a core dump.
Seen in the PR https://github.com/pantsbuild/pants/pull/11413 in CI.

```
thread panicked while processing panic. aborting.
491./build-support/githooks/pre-commit: line 53:  9562 Illegal instruction     (core dumped) ./pants --changed-since="${MERGE_BASE}" --tag='-nolint' lint
```

There is no core dump, though.
Hey folks: we're still investigating this one.

If you have a persistent Linux environment, getting set up to capture core dumps looks like:
```
echo '* soft core unlimited' | sudo tee -a /etc/security/limits.d/50-unlimited-core-dumps.conf
echo '* hard core unlimited' | sudo tee -a /etc/security/limits.d/50-unlimited-core-dumps.conf
echo 'kernel.core_pattern=/tmp/core-%e.%p.%h.%t' | sudo tee -a /etc/sysctl.d/99-core-dump-pattern.conf
# Ubuntu only: see https://unix.stackexchange.com/a/563505
sudo apt-get -y remove apport
```

Afterward, repros of this issue should include `(core dumped)`, and core files should go to `/tmp/core-*`. If you get one, stopping by [Slack](https://www.pantsbuild.org/docs/community#slack) to discuss it would be very appreciated!
So, digging a bit more, but my machine is configured to use systemd-coredump. I see this:
```
$ coredumpctl list --since=2021-01-22
TIME                            PID   UID   GID SIG COREFILE  EXE
Sat 2021-01-23 21:57:59 MST   30643  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:58:25 MST   34989  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:58:29 MST   35070  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:58:38 MST   35846  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:58:40 MST   36175  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:58:40 MST   37127  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:58:50 MST   38250  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:58:50 MST   37603  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:58:50 MST   38331  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:59:14 MST   42260  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:59:22 MST   45433  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:59:30 MST   45897  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:59:39 MST   48455  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:59:40 MST   48463  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:59:49 MST   50553  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 21:59:58 MST   53108  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:00:00 MST   53161  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:00:08 MST   56387  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:00:16 MST   58435  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:00:24 MST   59334  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:00:35 MST   62407  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:00:37 MST   62504  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:00:41 MST   63846  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:00:43 MST   63933  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:00:51 MST   64791  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:00:52 MST   64837  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:01:06 MST   67622  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:01:06 MST   65562  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:01:15 MST   70032  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:01:22 MST   70226  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:01:24 MST   72251  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:01:34 MST   73073  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:01:36 MST   72771  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:01:43 MST   73048  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:01:45 MST   76093  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:01:55 MST   77801  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:02:01 MST   77873  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:02:05 MST   82849  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:02:22 MST   85003  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:02:55 MST   99684  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:03:03 MST  101425  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:03:16 MST   36439  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:03:19 MST  105926  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:03:20 MST  108651  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:03:29 MST  110957  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:03:31 MST  111485  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:03:37 MST  112189  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:03:42 MST  112358  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:03:50 MST  113725  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:04:00 MST  114675  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:04:08 MST  112164  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:04:19 MST  117432  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:04:31 MST  119014  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:04:50 MST  126531  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:04:52 MST  123945  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:04:59 MST  129248  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:05:10 MST  132678  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:05:18 MST  133936  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:05:23 MST  136406  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:05:27 MST  137085  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:05:28 MST  137157  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:05:33 MST  137219  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:05:37 MST  137949  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:05:40 MST  138585  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:05:41 MST  137212  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:05:44 MST  138653  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:05:47 MST  138730  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:05:50 MST  138769  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:05:54 MST  138813  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:05:59 MST  138693  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:06:03 MST  138951  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:06:19 MST  138990  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:06:25 MST  139913  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:12:27 MST  141044  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:12:31 MST  143376  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:12:36 MST  144037  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:13:38 MST  155317  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:13:42 MST  155966  1000  1000   6 present   /usr/bin/python3.7
Sat 2021-01-23 22:13:47 MST  156613  1000  1000   6 present   /usr/bin/python3.7
```

I can assure you I saw nowhere near enough failures running Pants to account for that in the obvious way.
Looking at the most recent:
```
$ coredumpctl info 156613
           PID: 156613 (pantsd [/tmp/pr)
           UID: 1000 (jsirois)
           GID: 1000 (jsirois)
        Signal: 6 (ABRT)
     Timestamp: Sat 2021-01-23 22:13:46 MST (1 day 11h ago)
  Command Line: pantsd [/tmp/process-execution9G2PYb]
    Executable: /usr/bin/python3.7
 Control Group: /user.slice/user-1000.slice/user@1000.service/app.slice/vte-spawn-e0c78a26-9744-4fca-a345-804b9abbc658.scope
          Unit: user@1000.service
     User Unit: vte-spawn-e0c78a26-9744-4fca-a345-804b9abbc658.scope
         Slice: user-1000.slice
     Owner UID: 1000 (jsirois)
       Boot ID: 084d4d4aac3b4390810faba6fbb525d9
    Machine ID: c8b6b48c23a840ee862c8d2983461941
      Hostname: gill
       Storage: /var/lib/systemd/coredump/core.pantsd\x20[\x2ftmp\x2fpr.1000.084d4d4aac3b4390810faba6fbb525d9.156613.1611465226000000.zst
       Message: Process 156613 (pantsd [/tmp/pr) of user 1000 dumped core.
                
                Stack trace of thread 156615:
                #0  0x00007f1e41a91615 raise (libc.so.6 + 0x3d615)
                #1  0x00007f1e41a916a0 __restore_rt (libc.so.6 + 0x3d6a0)
                #2  0x00007f1e41a91615 raise (libc.so.6 + 0x3d615)
                #3  0x00007f1e41a7a862 abort (libc.so.6 + 0x26862)
                #4  0x00007f1e41ad35e8 __libc_message (libc.so.6 + 0x7f5e8)
                #5  0x00007f1e41ad361a __libc_fatal (libc.so.6 + 0x7f61a)
                #6  0x00007f1e41a44ae4 unwind_cleanup (libpthread.so.0 + 0x12ae4)
                #7  0x00007f1e400a20bf n/a (/tmp/process-execution9G2PYb/src/python/pants/engine/internals/native_engine.so + 0xa7e0bf)
```

The process execution dir for the (clearly) integration test is gone; so 1st:
```
$ mkdir -p `dirname /tmp/process-execution9G2PYb/src/python/pants/engine/internals/native_engine.so`
$ cp src/python/pants/engine/internals/native_engine.so /tmp/process-execution9G2PYb/src/python/pants/engine/internals/native_engine.so
```

And I find:
```
$ coredumpctl debug 156613
           PID: 156613 (pantsd [/tmp/pr)
           UID: 1000 (jsirois)
           GID: 1000 (jsirois)
        Signal: 6 (ABRT)
     Timestamp: Sat 2021-01-23 22:13:46 MST (1 day 11h ago)
  Command Line: pantsd [/tmp/process-execution9G2PYb]
    Executable: /usr/bin/python3.7
 Control Group: /user.slice/user-1000.slice/user@1000.service/app.slice/vte-spawn-e0c78a26-9744-4fca-a345-804b9abbc658.scope
          Unit: user@1000.service
     User Unit: vte-spawn-e0c78a26-9744-4fca-a345-804b9abbc658.scope
         Slice: user-1000.slice
     Owner UID: 1000 (jsirois)
       Boot ID: 084d4d4aac3b4390810faba6fbb525d9
    Machine ID: c8b6b48c23a840ee862c8d2983461941
      Hostname: gill
       Storage: /var/lib/systemd/coredump/core.pantsd\x20[\x2ftmp\x2fpr.1000.084d4d4aac3b4390810faba6fbb525d9.156613.1611465226000000.zst
       Message: Process 156613 (pantsd [/tmp/pr) of user 1000 dumped core.
                
                Stack trace of thread 156615:
                #0  0x00007f1e41a91615 raise (libc.so.6 + 0x3d615)
                #1  0x00007f1e41a916a0 __restore_rt (libc.so.6 + 0x3d6a0)
                #2  0x00007f1e41a91615 raise (libc.so.6 + 0x3d615)
                #3  0x00007f1e41a7a862 abort (libc.so.6 + 0x26862)
                #4  0x00007f1e41ad35e8 __libc_message (libc.so.6 + 0x7f5e8)
                #5  0x00007f1e41ad361a __libc_fatal (libc.so.6 + 0x7f61a)
                #6  0x00007f1e41a44ae4 unwind_cleanup (libpthread.so.0 + 0x12ae4)
                #7  0x00007f1e400a20bf n/a (/tmp/process-execution9G2PYb/src/python/pants/engine/internals/native_engine.so + 0xa7e0bf)

GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/python3.7...
(No debugging symbols found in /usr/bin/python3.7)
[New LWP 156615]
[New LWP 156614]
[New LWP 156622]
[New LWP 156616]
[New LWP 156623]
[New LWP 156617]
[New LWP 156638]
[New LWP 156624]
[New LWP 156640]
[New LWP 156629]
[New LWP 156613]
[New LWP 156633]
[New LWP 156631]
[New LWP 156636]
[New LWP 156628]
[New LWP 156637]
[New LWP 156634]
[New LWP 156632]
[New LWP 156639]
[New LWP 156630]
[New LWP 156635]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
--Type <RET> for more, q to quit, c to continue without paging--
Core was generated by `pantsd [/tmp/process-execution9G2PYb]                                         '.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f1e41a91615 in raise () from /usr/lib/libc.so.6
[Current thread is 1 (Thread 0x7f1e3e69f640 (LWP 156615))]
(gdb) info threads
  Id   Target Id                                    Frame 
* 1    Thread 0x7f1e3e69f640 (LWP 156615) (Exiting) 0x00007f1e41a91615 in raise () from /usr/lib/libc.so.6
  2    Thread 0x7f1e3e8a0640 (LWP 156614)           0x00007f1e41a419c8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  3    Thread 0x7f1e3e29d640 (LWP 156622)           0x00007f1e41a419c8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  4    Thread 0x7f1e3649e640 (LWP 156616)           0x00007f1e41b545de in epoll_wait () from /usr/lib/libc.so.6
  5    Thread 0x7f1e3e09c640 (LWP 156623)           0x00007f1e41a419c8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  6    Thread 0x7f1e3e49e640 (LWP 156617)           0x00007f1e41a416a2 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  7    Thread 0x7d66161f6640 (LWP 156638)           0x00007f1e41a419c8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  8    Thread 0x7f1e3de9b640 (LWP 156624)           0x00007f1e41a419c8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  9    Thread 0x7d6615df4640 (LWP 156640)           0x00007f1e41a419c8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  10   Thread 0x7d6617fff640 (LWP 156629)           0x00007f1e41b4ed5d in syscall () from /usr/lib/libc.so.6
  11   Thread 0x7f1e418dc740 (LWP 156613)           0x00007f1e41d107a5 in ?? () from /usr/lib/libpython3.7m.so.1.0
  12   Thread 0x7d6616bfb640 (LWP 156633)           0x00007f1e41a419c8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  13   Thread 0x7d66175fd640 (LWP 156631)           0x00007f1e41a44248 in do_futex_wait.constprop () from /usr/lib/libpthread.so.0
  14   Thread 0x7d66165f8640 (LWP 156636)           0x00007f1e41a419c8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  15   Thread 0x7f1e1c2d7640 (LWP 156628)           0x00007f1e41b545de in epoll_wait () from /usr/lib/libc.so.6
  16   Thread 0x7d66163f7640 (LWP 156637)           0x00007f1e41a419c8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  17   Thread 0x7d66169fa640 (LWP 156634)           0x00007f1e41a416a2 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  18   Thread 0x7d6616dfc640 (LWP 156632)           0x00007f1e41a419c8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  19   Thread 0x7d6615ff5640 (LWP 156639)           0x00007f1e41a419c8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  20   Thread 0x7d6617dfe640 (LWP 156630)           0x00007f1e41a416a2 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  21   Thread 0x7d66167f9640 (LWP 156635)           0x00007f1e41a419c8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
(gdb) bt
#0  0x00007f1e41a91615 in raise () from /usr/lib/libc.so.6
#1  <signal handler called>
#2  0x00007f1e41a91615 in raise () from /usr/lib/libc.so.6
#3  0x00007f1e41a7a862 in abort () from /usr/lib/libc.so.6
#4  0x00007f1e41ad35e8 in __libc_message () from /usr/lib/libc.so.6
#5  0x00007f1e41ad361a in __libc_fatal () from /usr/lib/libc.so.6
#6  0x00007f1e41a44ae4 in unwind_cleanup () from /usr/lib/libpthread.so.0
#7  0x00007f1e400a20bf in panic_unwind::real_imp::cleanup () at library/panic_unwind/src/gcc.rs:78
#8  panic_unwind::__rust_panic_cleanup () at library/panic_unwind/src/lib.rs:101
#9  0x00007f1e40090e0d in std::panicking::try::cleanup () at library/std/src/panicking.rs:362
#10 0x00007f1e3f7dbd46 in std::panicking::try::do_catch<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>> (data=<optimized out>, payload=0x7f1e3e69fcb0)
    at /home/jsirois/.rustup/toolchains/1.49.0-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:406
#11 std::panicking::try<core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>,std::panic::AssertUnwindSafe<closure-0>> (f=...) at /home/jsirois/.rustup/toolchains/1.49.0-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345
#12 std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>> (f=...) at /home/jsirois/.rustup/toolchains/1.49.0-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396
#13 tokio::runtime::task::harness::Harness<core::future::from_generator::GenFuture<generator-0>, alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>::poll<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>> (
    self=...) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/harness.rs:89
#14 tokio::runtime::task::raw::poll<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>> (ptr=...) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/raw.rs:104
#15 0x00007f1e3ffacc15 in tokio::runtime::task::raw::RawTask::poll (self=...) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/raw.rs:66
#16 tokio::runtime::task::Notified<alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>::run<alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>> (self=...) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/mod.rs:169
#17 tokio::runtime::thread_pool::worker::{{impl}}::run_task::{{closure}} () at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:349
#18 tokio::coop::with_budget::{{closure}}<core::result::Result<alloc::boxed::Box<tokio::runtime::thread_pool::worker::Core, alloc::alloc::Global>, ()>,closure-0> (cell=0x7f1e30000c8a) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/coop.rs:127
#19 std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget>>::try_with<core::cell::Cell<tokio::coop::Budget>,closure-0,core::result::Result<alloc::boxed::Box<tokio::runtime::thread_pool::worker::Core, alloc::alloc::Global>, ()>> (f=..., self=<optimized out>)
    at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/thread/local.rs:272
#20 std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget>>::with<core::cell::Cell<tokio::coop::Budget>,closure-0,core::result::Result<alloc::boxed::Box<tokio::runtime::thread_pool::worker::Core, alloc::alloc::Global>, ()>> (f=..., self=<optimized out>)
    at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/thread/local.rs:248
#21 tokio::coop::with_budget<core::result::Result<alloc::boxed::Box<tokio::runtime::thread_pool::worker::Core, alloc::alloc::Global>, ()>,closure-0> (budget=..., f=...) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/coop.rs:120
#22 tokio::coop::budget<core::result::Result<alloc::boxed::Box<tokio::runtime::thread_pool::worker::Core, alloc::alloc::Global>, ()>,closure-0> (f=...) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/coop.rs:96
#23 tokio::runtime::thread_pool::worker::Context::run_task (self=0x7f1e3e69eb00, task=..., core=<optimized out>) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:348
#24 0x00007f1e3ffab8c5 in tokio::runtime::thread_pool::worker::Context::run (self=0x7f1e3e69eb00, core=0x5633151e9ed0) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:320
#25 tokio::runtime::thread_pool::worker::run::{{closure}} () at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:305
#26 tokio::macros::scoped_tls::ScopedKey<tokio::runtime::thread_pool::worker::Context>::set<tokio::runtime::thread_pool::worker::Context,closure-0,()> (t=0x7f1e3e69eb00, f=..., self=<optimized out>)
    at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/macros/scoped_tls.rs:63
#27 tokio::runtime::thread_pool::worker::run (worker=...) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:302
#28 0x00007f1e3ffa52c0 in tokio::runtime::thread_pool::worker::{{impl}}::launch::{{closure}} () at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/thread_pool/worker.rs:281
#29 tokio::runtime::blocking::task::{{impl}}::poll<closure-0,()> (self=..., _cx=<optimized out>) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/blocking/task.rs:41
#30 tokio::runtime::task::core::{{impl}}::poll::{{closure}}<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule> (ptr=0x563314cccee0)
    at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/core.rs:173
#31 tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<closure-0>>>::with_mut<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<closure-0>>,core::task::poll::Poll<()>,closure-0> (
--Type <RET> for more, q to quit, c to continue without paging--
    self=0x563314cccee0, f=...) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/loom/std/unsafe_cell.rs:14
#32 tokio::runtime::task::core::Core<tokio::runtime::blocking::task::BlockingTask<closure-0>, tokio::runtime::blocking::schedule::NoopSchedule>::poll<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule> (self=0x563314cccee0, 
    header=<optimized out>) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/core.rs:158
#33 tokio::runtime::task::harness::{{impl}}::poll::{{closure}}<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule> () at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/harness.rs:107
#34 core::ops::function::FnOnce::call_once<closure-0,()> () at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/core/src/ops/function.rs:227
#35 std::panic::{{impl}}::call_once<core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>,closure-0> (self=..., _args=<optimized out>) at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/panic.rs:322
#36 std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>> (data=<optimized out>) at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/panicking.rs:381
#37 std::panicking::try<core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>,std::panic::AssertUnwindSafe<closure-0>> (f=...) at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/panicking.rs:345
#38 std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>> (f=...) at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/panic.rs:396
#39 tokio::runtime::task::harness::Harness<tokio::runtime::blocking::task::BlockingTask<closure-0>, tokio::runtime::blocking::schedule::NoopSchedule>::poll<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule> (self=...)
    at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/harness.rs:89
#40 tokio::runtime::task::raw::poll<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule> (ptr=...) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/raw.rs:104
#41 0x00007f1e3ff854f1 in tokio::runtime::task::raw::RawTask::poll (self=...) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/raw.rs:66
#42 tokio::runtime::task::Notified<tokio::runtime::blocking::schedule::NoopSchedule>::run<tokio::runtime::blocking::schedule::NoopSchedule> (self=...) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/task/mod.rs:169
#43 tokio::runtime::blocking::pool::Inner::run (self=0x5633151aed20, worker_id=1) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/blocking/pool.rs:250
#44 tokio::runtime::blocking::pool::{{impl}}::spawn_thread::{{closure}}::{{closure}} () at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/blocking/pool.rs:230
#45 tokio::runtime::context::enter<closure-0,()> (new=..., f=...) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/context.rs:72
#46 tokio::runtime::handle::Handle::enter<closure-0,()> (self=0x7f1e3e69ed10, f=...) at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/handle.rs:76
#47 tokio::runtime::blocking::pool::{{impl}}::spawn_thread::{{closure}} () at /home/jsirois/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.24/src/runtime/blocking/pool.rs:229
#48 std::sys_common::backtrace::__rust_begin_short_backtrace<closure-0,()> (f=...) at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/sys_common/backtrace.rs:125
#49 0x00007f1e3ff87df9 in std::thread::{{impl}}::spawn_unchecked::{{closure}}::{{closure}}<closure-0,()> () at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/thread/mod.rs:474
#50 std::panic::{{impl}}::call_once<(),closure-0> (self=<error reading variable: Cannot access memory at address 0x0>, _args=<optimized out>) at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/panic.rs:322
#51 std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure-0>,()> (data=<optimized out>) at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/panicking.rs:381
#52 std::panicking::try<(),std::panic::AssertUnwindSafe<closure-0>> (f=<error reading variable: Cannot access memory at address 0x0>) at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/panicking.rs:345
#53 std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure-0>,()> (f=<error reading variable: Cannot access memory at address 0x0>) at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/panic.rs:396
#54 std::thread::{{impl}}::spawn_unchecked::{{closure}}<closure-0,()> () at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/thread/mod.rs:473
#55 core::ops::function::FnOnce::call_once<closure-0,()> () at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/core/src/ops/function.rs:227
#56 0x00007f1e4009956a in alloc::boxed::{{impl}}::call_once<(),FnOnce<()>,alloc::alloc::Global> () at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/alloc/src/boxed.rs:1307
#57 alloc::boxed::{{impl}}::call_once<(),alloc::boxed::Box<FnOnce<()>, alloc::alloc::Global>,alloc::alloc::Global> () at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/alloc/src/boxed.rs:1307
#58 std::sys::unix::thread::{{impl}}::new::thread_start () at library/std/src/sys/unix/thread.rs:71
#59 0x00007f1e41a3b3e9 in start_thread () from /usr/lib/libpthread.so.0
#60 0x00007f1e41b54293 in clone () from /usr/lib/libc.so.6
(gdb) q
```

Let me know if you need any of the 79+ core dumps I have.

@jsirois : Awesome, thanks! One thing to check would be to sample the crashes to confirm that they have relatively consistent stacks. Then, if you're able to [file with tokio](https://github.com/tokio-rs/tokio/issues/new/choose), that would be super helpful. They may tell us "we can't help with this until you're on 1.0.x", but we can point to [our blocker](https://github.com/tokio-rs/tokio/issues/2965#issuecomment-763884098) there.
OK, this is _extremely_ easy to repro. I just run `./pants fmt lint typecheck test src/python:: tests/python::` and out plop 70+ more core dumps. In my most recent test, of 71 dumps, 70 had the same origination address in native_engine.so. Sampling 5 random processes of those 70 gave the exact same traceback line for line varying only in addresses and that traceback is the same mod addresses as the one above.

@stuhood I'm hesitant to file against tokio since I don't know enough here to assign blame. I do see that the backtrace only has stdlib and tokio lines, but does that preclude us having used the library badly? Presumably you have the confidence and correct words to file the issue given I've confirmed the backtrace is repeatable and unchanging.
Filed as #3499. Will see if I can repro.
Ah, yea: I repro that particular crash under vagrant in an Ubuntu image.

Hopefully it is related to the foreground crash in some way.

Resolving patch:
diff --git a/src/python/pants/init/logging.py b/src/python/pants/init/logging.py
--- a/src/python/pants/init/logging.py
+++ b/src/python/pants/init/logging.py
@@ -9,6 +9,7 @@
 from contextlib import contextmanager
 from io import BufferedReader, TextIOWrapper
 from logging import Formatter, LogRecord, StreamHandler
+from pathlib import PurePath
 from typing import Dict, Iterator
 
 import pants.util.logging as pants_logging
@@ -139,7 +140,7 @@ def initialize_stdio(global_bootstrap_options: OptionValueContainer) -> Iterator
     deprecated_log_path = os.path.join(
         global_bootstrap_options.pants_workdir, "pantsd", "pantsd.log"
     )
-    log_path = os.path.join(global_bootstrap_options.pants_workdir, "pants.log")
+    log_path = str(pants_log_path(PurePath(global_bootstrap_options.pants_workdir)))
     safe_mkdir_for(deprecated_log_path)
     safe_mkdir_for(log_path)
     # NB: We append to the deprecated log location with a deprecated conditional that never
@@ -182,6 +183,11 @@ def initialize_stdio(global_bootstrap_options: OptionValueContainer) -> Iterator
         sys.__stdin__, sys.__stdout__, sys.__stderr__ = sys.stdin, sys.stdout, sys.stderr
 
 
+def pants_log_path(workdir: PurePath) -> PurePath:
+    """Given the path of the workdir, returns the `pants.log` path."""
+    return workdir / "pants.log"
+
+
 def _get_log_levels_by_target(
     global_bootstrap_options: OptionValueContainer,
 ) -> Dict[str, LogLevel]:
diff --git a/src/python/pants/pantsd/pants_daemon.py b/src/python/pants/pantsd/pants_daemon.py
--- a/src/python/pants/pantsd/pants_daemon.py
+++ b/src/python/pants/pantsd/pants_daemon.py
@@ -8,6 +8,7 @@
 import sys
 import time
 import warnings
+from pathlib import PurePath
 from typing import Any
 
 from setproctitle import setproctitle as set_process_title
@@ -18,7 +19,7 @@
 from pants.engine.environment import CompleteEnvironment
 from pants.engine.internals import native_engine
 from pants.init.engine_initializer import GraphScheduler
-from pants.init.logging import initialize_stdio
+from pants.init.logging import initialize_stdio, pants_log_path
 from pants.init.util import init_workdir
 from pants.option.global_options import GlobalOptions
 from pants.option.option_value_container import OptionValueContainer
@@ -30,6 +31,7 @@
 from pants.pantsd.service.scheduler_service import SchedulerService
 from pants.pantsd.service.store_gc_service import StoreGCService
 from pants.util.contextutil import argv_as, hermetic_environment_as
+from pants.util.dirutil import safe_open
 from pants.util.logging import LogLevel
 from pants.util.strutil import ensure_text
 
@@ -131,14 +133,29 @@ def __init__(
 
         self._logger = logging.getLogger(__name__)
 
-    @staticmethod
-    def _close_stdio():
-        """Close stdio streams to avoid output in the tty that launched pantsd."""
-        for fd in (sys.stdin, sys.stdout, sys.stderr):
-            file_no = fd.fileno()
+    def _close_stdio(self, log_path: PurePath):
+        """Close stdio and append to a log path instead.
+
+        The vast majority of Python-level IO will be re-routed to thread-local destinations by
+        `initialize_stdio`, but we close stdio to avoid any stray output in the tty that launched
+        pantsd.
+
+        Rather than leaving 0, 1, 2 dangling though, we open replacements as a backstop for fatal
+        errors or unmodified code (such as Rust panic handlers) that might expect them to be valid
+        file handles.
+        """
+        for attr, writable in (("stdin", False), ("stdout", True), ("stderr", True)):
+            # Close the old.
+            fd = getattr(sys, attr)
+            fileno = fd.fileno()
             fd.flush()
             fd.close()
-            os.close(file_no)
+
+            # Open the new.
+            temp_fd = safe_open(log_path, "w") if writable else open(os.devnull)
+            os.dup2(temp_fd.fileno(), fileno)
+            setattr(sys, attr, os.fdopen(fileno, mode=("w" if writable else "r")))
+        sys.__stdin__, sys.__stdout__, sys.__stderr__ = sys.stdin, sys.stdout, sys.stderr
 
     def _initialize_metadata(self) -> None:
         """Writes out our pid and other metadata.
@@ -165,7 +182,7 @@ def run_sync(self):
 
         # Switch log output to the daemon's log stream, and empty `env` and `argv` to encourage all
         # further usage of those variables to happen via engine APIs and options.
-        self._close_stdio()
+        self._close_stdio(pants_log_path(PurePath(global_bootstrap_options.pants_workdir)))
         with initialize_stdio(global_bootstrap_options), argv_as(
             tuple()
         ), hermetic_environment_as():


Test patch: