Repository: meetecho/janus-gateway

Issue #154: eventfd spinning
Created: 2015-02-13T17:47:31Z
I figured out what was sometimes making threads "spin", consuming as much cpu as they can: write()s to eventfd.
First I identified a thread that was "going crazy", and then counted syscalls with strace over a few seconds:
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 59.84    0.712945           6    110297           write
 19.47    0.231949           6     36765           clock_gettime
 10.60    0.126276           7     18383           poll
  9.95    0.118586           6     18382           read
  0.14    0.001683          12       135        32 futex
------ ----------- ----------- --------- --------- ----------------
100.00    1.191439                183962        32 total

that seems like more write() than makes sense for sending rtp/srtp packets. Let's see what it's writing...
clock_gettime(CLOCK_MONOTONIC, {823252, 946459022}) = 0
poll([{fd=15, events=POLLIN}, {fd=26, events=POLLIN}, {fd=28, events=POLLIN}, {fd=32, events=POLLIN}, {fd=34, events=POLLIN}, {fd=25, events=POLLIN}, {fd=27, events=POLLIN}, {fd=31, events=POLLIN}, {fd=33, events=POLLIN}], 9, 2553) = 4 ([{fd=15, revents=POLLIN}, {fd=27, revents=POLLIN}, {fd=31, revents=POLLIN}, {fd=33, revents=POLLIN}])
read(15, "\6\0\0\0\0\0\0\0", 16)        = 8
clock_gettime(CLOCK_MONOTONIC, {823252, 946602023}) = 0
write(15, "\1\0\0\0\0\0\0\0", 8)        = 8
write(15, "\1\0\0\0\0\0\0\0", 8)        = 8
write(15, "\1\0\0\0\0\0\0\0", 8)        = 8
write(15, "\1\0\0\0\0\0\0\0", 8)        = 8
write(15, "\1\0\0\0\0\0\0\0", 8)        = 8
write(15, "\1\0\0\0\0\0\0\0", 8)        = 8
clock_gettime(CLOCK_MONOTONIC, {823252, 946880949}) = 0
poll([{fd=15, events=POLLIN}, {fd=26, events=POLLIN}, {fd=28, events=POLLIN}, {fd=32, events=POLLIN}, {fd=34, events=POLLIN}, {fd=25, events=POLLIN}, {fd=27, events=POLLIN}, {fd=31, events=POLLIN}, {fd=33, events=POLLIN}], 9, 2553) = 4 ([{fd=15, revents=POLLIN}, {fd=27, revents=POLLIN}, {fd=31, revents=POLLIN}, {fd=33, revents=POLLIN}])
read(15, "\6\0\0\0\0\0\0\0", 16)        = 8
clock_gettime(CLOCK_MONOTONIC, {823252, 947022532}) = 0
write(15, "\1\0\0\0\0\0\0\0", 8)        = 8
write(15, "\1\0\0\0\0\0\0\0", 8)        = 8
write(15, "\1\0\0\0\0\0\0\0", 8)        = 8
write(15, "\1\0\0\0\0\0\0\0", 8)        = 8
write(15, "\1\0\0\0\0\0\0\0", 8)        = 8
write(15, "\1\0\0\0\0\0\0\0", 8)        = 8
clock_gettime(CLOCK_MONOTONIC, {823252, 947316721}) = 0

So it reads 8 bytes from fd 15, and then writes 8 bytes to fd 15, 6 times. and the data is 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 each time. (and it does this rapidly and continuously)
root@janus02:~# ls -l /proc/615/fd
total 0
lrwx------ 1 root root 64 Feb 13 17:38 0 -> /dev/null
l-wx------ 1 root root 64 Feb 13 17:38 1 -> /opt/log/janus/janus.log
lrwx------ 1 root root 64 Feb 13 17:38 10 -> socket:[5925975]
lrwx------ 1 root root 64 Feb 13 17:38 11 -> anon_inode:[eventfd]
lrwx------ 1 root root 64 Feb 13 17:38 12 -> anon_inode:[eventfd]
lrwx------ 1 root root 64 Feb 13 17:38 13 -> socket:[5924500]
lrwx------ 1 root root 64 Feb 13 17:38 14 -> socket:[5924501]
lrwx------ 1 root root 64 Feb 13 17:38 15 -> anon_inode:[eventfd]
lrwx------ 1 root root 64 Feb 13 17:38 16 -> socket:[5925728]
...

fd 15 is from eventfd()
Maybe it's some libnice or glib thing... I'll try to get a backtrace of the relevant thread.

Duplicate issue #587: ICE setup fails due to socket descriptor leak
Created: 2016-07-26T11:53:45Z
I've recently encountered an issue caused by the underlying libnice library.
This can be reproduced easily by setting rtp_port_range to a small window, e.g. 60000-60001, and connecting to Janus VideoRoom multiple times.
Here is my janus.cfg: http://pastebin.com/mkk8hDik
I have libnice 0.1.13 on ArchLinux.
What happens is the ICE agent attempts to bind a passive TCP socket on each discovery for local candidates. These TCP socket descriptors are not properly closed in 0.1.13 of libnice. If all ports are occupied, the discovery process fails and Janus will no longer be able to create new peer connections.
This issue was fixed here: https://phabricator.freedesktop.org/D228, but not included in the 0.1.13 libnice release. I have verified this fixed the hanging issue for Janus by checking out libnice 0.1.13 tag and cherry-picking the specific diff (somehow Janus is not working when compiled with the most recent libnice master branch).
Unfortunately, I did not have time to dig deeper into Janus code to see how to address this from higher level code. However, I hope this may be of help to anyone having similar issues.

Closing comment (2016-07-27T12:49:38Z): @lminiero Sorry for the late reply.
Yes, I'm only getting this with TCP candidates. I've seen the leak happen just once with UDP candidates, but not reproducible, I'll file a report if I can find a way to reproduce it. I'm not sure if this is directly related to #154. An example of TCP fd leak during candidate discovery:
[pid 19382] socket(PF_NETLINK, SOCK_RAW, 0) = 21
[pid 19382] bind(21, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
[pid 19382] close(21)                   = 0
[pid 19382] socket(PF_INET, SOCK_DGRAM|SOCK_CLOEXEC, IPPROTO_UDP) = 23
[pid 19382] bind(23, {sa_family=AF_INET, sin_port=htons(60001), sin_addr=inet_addr("10.140.0.2")}, 16) = 0
[pid 19382] socket(PF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_TCP) = 24
[pid 19382] bind(24, {sa_family=AF_INET, sin_port=htons(60000), sin_addr=inet_addr("10.140.0.2")}, 16) = -1 EADDRINUSE (Address already in use)
[pid 19382] close(24)                   = 0
[pid 19382] socket(PF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_TCP) = 24
[pid 19382] bind(24, {sa_family=AF_INET, sin_port=htons(60001), sin_addr=inet_addr("10.140.0.2")}, 16) = 0
[pid 19384] +++ exited with 0 +++
[pid 19343] close(23)                   = 0
[pid 19343] close(21)                   = 0
[pid 19383] +++ exited with 0 +++
[pid 19382] +++ exited with 0 +++
Process 19386 attached
Process 19387 attached
Process 19388 attached
[pid 19386] socket(PF_NETLINK, SOCK_RAW, 0) = 23
[pid 19386] bind(23, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
[pid 19386] close(23)                   = 0
[pid 19386] socket(PF_INET, SOCK_DGRAM|SOCK_CLOEXEC, IPPROTO_UDP) = 25
[pid 19386] bind(25, {sa_family=AF_INET, sin_port=htons(60001), sin_addr=inet_addr("10.140.0.2")}, 16) = 0
[pid 19386] socket(PF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_TCP) = 26
[pid 19386] bind(26, {sa_family=AF_INET, sin_port=htons(60001), sin_addr=inet_addr("10.140.0.2")}, 16) = -1 EADDRINUSE (Address already in use)
[pid 19386] close(26)                   = 0
[pid 19386] socket(PF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_TCP) = 26
[pid 19386] bind(26, {sa_family=AF_INET, sin_port=htons(60000), sin_addr=inet_addr("10.140.0.2")}, 16) = -1 EADDRINUSE (Address already in use)
[pid 19386] close(26)                   = 0
[pid 19386] close(25)                   = 0

We can see that file descriptor 24 is not closed and continues to occupy port 60001, and next bind attempt fails since all ports in range 60000-60001 are occupied. On Janus side:
Creating new session: 1347208485898656
Creating new handle in session 1347208485898656: 3605334577391632
[ERR] [janus.c:janus_plugin_handle_sdp:2632] [5164479381175082] Error gathering candidates!
[ERR] [janus.c:janus_plugin_push_event:2513] [5164479381175082] Cannot push event (handle not available anymore or negotiation stopped)
[3605334577391632] Creating ICE agent (ICE Lite mode, controlled)
(process:10): libnice-DEBUG: Created NiceStream (21 created, 19 destroyed)
(process:10): libnice-DEBUG: Created NiceComponent (21 created, 19 destroyed)
(process:10): libnice-DEBUG: Agent 0x7f1c1c00a5c0 : allocating stream id 1 (0x7f1c1c048000)
(process:10): libnice-DEBUG: Agent 0x7f1c1c00a5c0 : In ICE-LITE mode, starting candidate gathering.
(process:10): libnice-DEBUG: Agent 0x7f1c1c00a5c0 : libnice compiled without UPnP support
(process:10): libnice-DEBUG: Agent 0x7f1c1c00a5c0: Trying to create host candidate on port 60000
(process:10): libnice-DEBUG: Agent 0x7f1c1c00a5c0: Could not set IPV6 socket ToS: Protocol not available
(process:10): libnice-DEBUG: Component 0x7f1c1c04a400 (agent 0x7f1c1c00a5c0): Attach source (stream 1).
(process:10): libnice-DEBUG: Attaching source 0x7f1c1c03d200 (socket 0x7f1c1c03f520, FD 46) to context 0x7f1c1c03d000
(process:10): libnice-DEBUG: Agent 0x7f1c1c00a5c0: Trying to create host candidate on port 60001
(process:10): libnice-DEBUG: Agent 0x7f1c1c00a5c0: Trying to create host candidate on port 60001
(process:10): libnice-DEBUG: Agent 0x7f1c1c00a5c0: Trying to create host candidate on port 60000
(process:10): libnice-DEBUG: Agent 0x7f1c1c00a5c0: Unable to add local host candidate 10.140.0.2 for s1:1. Invalid interface?
(process:10): libnice-DEBUG: Free socket sources for component 0x7f1c1c04a400.
(process:10): libnice-DEBUG: Detaching source 0x7f1c1c03d200 (socket 0x7f1c1c03f520, FD 46) from context 0x7f1c1c03d000
Cleaning up handle 5164479381175082...
(process:10): libnice-DEBUG: Free socket sources for component 0x7f1c1c043fa0.
(process:10): libnice-DEBUG: Destroyed NiceComponent (21 created, 20 destroyed)
(process:10): libnice-DEBUG: Destroyed NiceStream (21 created, 20 destroyed)
[5164479381175082] WebRTC resources freed
[5164479381175082] Handle and related resources freed

Since there's not much we can do from application level, should we close this issue, and leave it for future reference?