build した beam で iex/mix を実行、coredump を解析

### clone 

https://github.com/erlang/otp.git は `~/Repos/otp` にクローン済みでスタート

``` 
git clone ~/Repos/otp otp-26.1.2 -b OTP-26.1.2
cd otp-26.1.2
```
### configure

```
export ERL_TOP=`pwd`
# ./configure -h で オプションは確認できる
./configure --enable-kernel-poll --enable-dirty-schedulers --enable-jit --enable-debug-mode
```

### make 

- [Compile Server](https://www.erlang.org/doc/man/erlc#compile-server)
- CPU 全部 `-j`

を使って make

```
ERLC_USE_SERVER=true make -j
```

#### beam.debug.smp の make

```
make -j -C erts/emulator debug
```

### 生成物の確認

```
ls bin/
cerl    dialyzer  erl_call  escript             no_dot_erlang.script  start.script      start_clean.script  start_sasl.script  x86_64-pc-linux-gnu
ct_run  erl       erlc      no_dot_erlang.boot  start.boot            start_clean.boot  start_sasl.boot     typer
```

```
ls bin/x86_64-pc-linux-gnu/
beam.debug.jit  beam.debug.smp  beam.jit  beam.smp  ct_run  dialyzer  dyn_erl  epmd  erl_call  erl_child_setup  erl_child_setup.debug  erlc  erlexec  escript  heart  inet_gethost  jit-reader.so  libbeam.a  run_erl  to_erl  typer
```

### iex コマンドを確かめる

```
ELIXIR_CLI_DRY_RUN=1 iex
erl -noshell -elixir_root /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib -pa /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib/elixir/ebin -s elixir start_iex -elixir ansi_enabled true -user elixir -extra --no-halt +iex
```

### build した beam.smp で iex を実行する

```
./bin/erl -noshell -elixir_root /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib -pa /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib/elixir/ebin -s elixir start_iex -elixir ansi_enabled true -user elixir -extra --no-halt +iex

Erlang/OTP 26 [erts-14.1.1] [source] [64-bit] [smp:32:32] [ds:32:32:10] [async-threads:1] [jit:ns]

Interactive Elixir (1.15.7) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
```

### iex -S mix の実行でエラー & 回避

#### エラー

```
/home/pojiro/Sandbox/otp-26.1.2/bin/erl -noshell -elixir_root /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib -pa /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib/elixir/ebin -s elixir start_iex -elixir ansi_enabled true -user elixir -extra --no-halt +iex -S mix
** (SyntaxError) /home/pojiro/.asdf/shims/mix:6:19: syntax error before: '.'
    |
  6 | exec /home/pojiro/.asdf/bin/asdf exec "mix" "$@" # asdf_allow: ' asdf '
    |                   ^
    (elixir 1.15.7) lib/code.ex:1435: Code.require_file/2
```

#### 回避

mix をフルパスで与えると回避できるが、原因不明

ref. https://github.com/asdf-vm/asdf/issues/926#issuecomment-1890830438

```
/home/pojiro/Sandbox/otp-26.1.2/bin/erl -noshell -elixir_root /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib -pa /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib/elixir/ebin -s elixir start_iex -elixir ansi_enabled true -user elixir -extra --no-halt +iex -S /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/mix
Erlang/OTP 26 [erts-14.1.1] [source] [64-bit] [smp:32:32] [ds:32:32:10] [async-threads:1] [jit:ns]

Interactive Elixir (1.15.7) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> 
```

### beam の coredump をとって、 gdb で解析できるようにする

前提： mix test が core を吐く、そのプロジェクトのディレクトリには cd 済み

#### コマンドを調べる

```
ELIXIR_CLI_DRY_RUN=1 mix test test/test_defer_load_nif_test.exs:10
erl -noshell -elixir_root /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib -pa /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib/elixir/ebin -s elixir start_cli -elixir ansi_enabled true -extra /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/mix test test/test_defer_load_nif_test.exs:10
```

#### build した beam.smp で実行する

```
/home/pojiro/Sandbox/otp-26.1.2/bin/erl -noshell -elixir_root /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib -pa /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib/elixir/ebin -s elixir start_iex -elixir ansi_enabled true -extra /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/mix test test/test_defer_load_nif_test.exs:10
...
Segmentation fault (コアダンプ)
```

#### build した beam.debug.smp で実行する

`erl` -> `cerl -debug` 点に注意

```
/home/pojiro/Sandbox/otp-26.1.2/bin/cerl -debug -noshell -elixir_root /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib -pa /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/../lib/elixir/ebin -s elixir start_iex -elixir ansi_enabled true -extra /home/pojiro/.asdf/installs/elixir/1.15.7-otp-26/bin/mix test test/test_defer_load_nif_test.exs:10
...
# コア吐かなくなった。。ぬー
```

#### コアを探す

```
cat /proc/sys/kernel/core_pattern 
|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h
```

systemd-coredump にパイプされているので、`/var/lib/systemd/coredump/` で探す。

```
ls -la /var/lib/systemd/coredump/
合計 83244
drwxr-xr-x   2 root root     4096  1月 16 20:34  .
drwxr-xr-x  12 root root     4096  1月  3 21:16  ..
-rw-r-----+  1 root root 17119855  1月 16 20:22 'core.beam\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.5858.1705404135000000.zst'
-rw-r-----+  1 root root 16890586  1月 16 20:24 'core.beam\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.6757.1705404273000000.zst'
-rw-r-----+  1 root root 16880366  1月 16 20:25 'core.beam\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.7049.1705404310000000.zst'
-rw-r-----+  1 root root 16995351  1月 16 20:25 'core.beam\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.7377.1705404313000000.zst'
-rw-r-----+  1 root root 17315907  1月 16 20:34 'core.beam\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.8778.1705404843000000.zst'
```

#### zstを解凍

```
cp /var/lib/systemd/coredump/'core.beam\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.8778.1705404843000000.zst' .
zstd -d 'core.beam\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.8778.1705404843000000.zst'
```

#### cerl -rcore で gdb 起動

- beam.smp で実行したコアの解析は -debug オプション無しでやる
- beam.debug.smp で実行したコアの解析は -debug オプション有りでやる

記事を書くきっかけの issue は beam.debug.smp だとコアダンプしなかったので、 beam.smp で実行したコアの解析をする

```
/home/pojiro/Sandbox/otp-26.1.2/bin/cerl -rcore core.beam\\x2esmp.1000.56e029b33eec4decbe02e9ab0408bd01.8778.1705404843000000
```

起動すると、

```
Core was generated by `/home/pojiro/Sandbox/otp-26.1.2/bin/x86_64-pc-linux-gnu/beam.smp -- -root /home'.
Program terminated with signal SIGSEGV, Segmentation fault.

warning: Section `.reg-xstate/5449' in core file too small.
#0  0x00007fa935683cc8 in ?? ()
[Current thread is 1 (Thread 0x7fa93251e640 (LWP 5449))]
%---------------------------------------------------------------------------
% Use etp-help for a command overview and general help.
%
% To use the Erlang support module, the environment variable ROOTDIR
% must be set to the toplevel installation directory of Erlang/OTP,
% so the etp-commands file becomes:
%     $ROOTDIR/erts/etc/unix/etp-commands
% Also, erl and erlc must be in the path.
%---------------------------------------------------------------------------
etp-set-max-depth 20
etp-set-max-string-length 100
--------------- System Information ---------------
OTP release: 26
ERTS version: 14.1.1
Arch: x86_64-pc-linux-gnu
Endianness: Little
Word size: 64-bit
BeamAsm support: yes
SMP support: yes
Thread support: yes
Kernel poll: Supported and used
Debug compiled: no
Lock checking: no
Lock counting: no
Node name: nonode@nohost
Number of schedulers: 32
Number of async-threads: 1
--------------------------------------------------
(gdb)
```


### 参考

- [Erlang/OTP ソースコードインストールのススメ](https://zenn.dev/voluntas/articles/erlang-source-code-install)
- [elixirの実行コマンドの実体を見る](https://qiita.com/ohr486/items/b8b8a93d2f8d52a6ce9b)
- [Erlangのビルドを速くしたい](https://qiita.com/ohr486/items/cfb520b8e6c74ad8e687)
- [2 Building and Installing Erlang/OTP](https://www.erlang.org/doc/installation_guide/install)
- [9 Debugging NIFs and Port Drivers](https://www.erlang.org/doc/tutorial/debugging)


### 読めてないけど

- [How to analyse a BEAM core dump?](https://www.erlang-solutions.com/blog/how-to-analyse-a-beam-core-dump/)