Repository: blakeblackshear/frigate

Issue:
Shared memory exception - FileExistsError: [Errno 17] File exists
**Describe the bug**
After certain period of time frigate pods gets into `CrashLoopBackOff` state. Problem gets resolved by manual pod deletion, pod restart doesn't help.

**Version of frigate**
0.8.4-5043040

**Config file**
```yaml
ffmpeg:
  global_args: -hide_banner -loglevel warning
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420p
logger:
  default: info
  logs:
    ffmpeg.front_garden.detect: debug
    frigate.video: debug
mqtt:
  host: mosquitto-nontls
  port: 1883
  user: frigate
  password: '{FRIGATE_MQTT_PASS}'
  stats_interval: 60
detectors:
  coral:
    type: edgetpu
    device: usb
cameras:
  front_garden:
    clips:
      enabled: true
    snapshots:
      enabled: true
      bounding_box: true
    ffmpeg:
      inputs:
        - path: rtsp://*****:{FRIGATE_RTSP_PASS}@nvr.host/channel1
          roles:
            - detect
            - rtmp
            - clips
    width: 640
    height: 480
    fps: 5
    motion:
      mask:
        - 596,0,410,0,418,150,377,231,502,270,597,213
        - 387,358,429,352,513,414,489,480,157,480,111,444,31,334,49,272,102,216,191,184,224,262,270,230,404,276
        - 220,22,218,68,0,74,0,23
  back_garden:
    clips:
      enabled: true
    snapshots:
      enabled: true
      bounding_box: true
    ffmpeg:
      inputs:
        - path: rtsp://*****:{FRIGATE_RTSP_PASS}@nvr.host/channel4
          roles:
            - detect
            - rtmp
            - clips
    width: 640
    height: 480
    fps: 5
    motion:
      mask:
        - 88,480,0,480,0,262,63,398,127,405,191,402,234,421,340,393,406,480
        - 577,259,539,290,500,271,474,217,488,161,544,153,571,187
        - 640,0,640,480,505,480,568,283,589,125,568,75,425,45,372,0
        - 0,0,205,0,138,27,0,74
  sidefront_garden:
    clips:
      enabled: true
    snapshots:
      enabled: true
      bounding_box: true
    ffmpeg:
      inputs:
        - path: rtsp://*****:{FRIGATE_RTSP_PASS}@nvr.host/channel2
          roles:
            - detect
            - rtmp
            - clips
    width: 480
    height: 640
    fps: 5
    motion:
      mask:
        - 103,107,415,104,448,0,0,0,0,154
  sideback_garden:
    clips:
      enabled: true
    snapshots:
      enabled: true
      bounding_box: true
    ffmpeg:
      inputs:
        - path: rtsp://*****:{FRIGATE_RTSP_PASS}@nvr.host/channel3
          roles:
            - detect
            - rtmp
            - clips
    width: 480
    height: 640
    fps: 5
    motion:
      mask:
        - 480,0,0,0,0,63,177,71,355,115,371,166,317,261,480,425

objects:
  track:
    - person
    - bicycle
```

**Frigate container logs**
```
 * Starting nginx nginx
   ...done.
frigate.app                    INFO    : Creating directory: /tmp/cache
Starting migrations
peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
peewee_migrate                 INFO    : There is nothing to migrate
frigate.mqtt                   INFO    : MQTT connected
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/frigate/frigate/__main__.py", line 15, in <module>
    frigate_app.start()
  File "/opt/frigate/frigate/app.py", line 225, in start
    self.start_detectors()
  File "/opt/frigate/frigate/app.py", line 149, in start_detectors
    shm_in = mp.shared_memory.SharedMemory(name=name, create=True, size=self.config.model.height*self.config.model.width*3)
  File "/usr/lib/python3.8/multiprocessing/shared_memory.py", line 100, in __init__
    self._fd = _posixshmem.shm_open(
FileExistsError: [Errno 17] File exists: '/front_garden'
```

**Frigate stats**
```json
{
  "back_garden": {
    "camera_fps": 4.6, 
    "capture_pid": 46, 
    "detection_fps": 0.0, 
    "pid": 40, 
    "process_fps": 4.6, 
    "skipped_fps": 0.0
  }, 
  "detection_fps": 0.0, 
  "detectors": {
    "coral": {
      "detection_start": 0.0, 
      "inference_speed": 10.0, 
      "pid": 36
    }
  }, 
  "front_garden": {
    "camera_fps": 4.5, 
    "capture_pid": 44, 
    "detection_fps": 0.0, 
    "pid": 39, 
    "process_fps": 4.5, 
    "skipped_fps": 0.0
  }, 
  "service": {
    "storage": {
      "/dev/shm": {
        "free": 4126.0, 
        "mount_type": "tmpfs", 
        "total": 4128.8, 
        "used": 2.7
      }, 
      "/media/frigate/clips": {
        "free": 121994.5, 
        "mount_type": "nfs4", 
        "total": 1081180.9, 
        "used": 958632.8
      }, 
      "/media/frigate/recordings": {
        "free": 121994.5, 
        "mount_type": "nfs4", 
        "total": 1081180.9, 
        "used": 958632.8
      }, 
      "/tmp/cache": {
        "free": 14861.0, 
        "mount_type": "overlay", 
        "total": 52576.1, 
        "used": 35014.0
      }
    }, 
    "uptime": 320, 
    "version": "0.8.4-5043040"
  }, 
  "sideback_garden": {
    "camera_fps": 5.0, 
    "capture_pid": 52, 
    "detection_fps": 0.0, 
    "pid": 42, 
    "process_fps": 5.0, 
    "skipped_fps": 0.0
  }, 
  "sidefront_garden": {
    "camera_fps": 4.6, 
    "capture_pid": 49, 
    "detection_fps": 0.0, 
    "pid": 41, 
    "process_fps": 4.6, 
    "skipped_fps": 0.0
  }
}
```

**FFprobe from your camera**

Run the following command and paste output below
```
ffprobe rtsp://*****:*****@nvr.host/channel1
ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
[rtsp @ 0x5600153ceb80] UDP timeout, retrying with TCP
[rtsp @ 0x5600153ceb80] method PAUSE failed: 404 File Not Found
[rtsp @ 0x5600153ceb80] Could not find codec parameters for stream 0 (Video: h264, none): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, rtsp, from 'rtsp://*****:*****@nvr.host/channel1':
  Metadata:
    title           : H264 Session
    comment         : H264 info
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264, none, 90k tbr, 90k tbn, 180k tbc
```

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Computer Hardware**
 - Platform: k3s
 - Node OS: Ubuntu 20.04.1 LTS
 - Install method: Helm Chart
 - Virtualization: None
 - Coral Version: USB 2.0
 - Network Setup: Wired

**Camera Info:**
 - Manufacturer: Hikvision
 - Model: DS-2CD2T65G1-I5
 - Resolution: 640x480
 - FPS: 30

**Additional context**
Same problem was observed with frigate version 0.8.1.


Discussion:

Resolving patch:
diff --git a/frigate/app.py b/frigate/app.py
--- a/frigate/app.py
+++ b/frigate/app.py
@@ -166,14 +166,23 @@ def start_detectors(self):
         model_shape = (self.config.model.height, self.config.model.width)
         for name in self.config.cameras.keys():
             self.detection_out_events[name] = mp.Event()
-            shm_in = mp.shared_memory.SharedMemory(
-                name=name,
-                create=True,
-                size=self.config.model.height * self.config.model.width * 3,
-            )
-            shm_out = mp.shared_memory.SharedMemory(
-                name=f"out-{name}", create=True, size=20 * 6 * 4
-            )
+
+            try:
+                shm_in = mp.shared_memory.SharedMemory(
+                    name=name, 
+                    create=True, 
+                    size=self.config.model.height*self.config.model.width * 3,
+                )
+            except FileExistsError:
+                shm_in = mp.shared_memory.SharedMemory(name=name)
+
+            try:
+                shm_out = mp.shared_memory.SharedMemory(
+                    name=f"out-{name}", create=True, size=20 * 6 * 4
+                )
+            except FileExistsError:
+                shm_out = mp.shared_memory.SharedMemory(name=f"out-{name}")
+
             self.detection_shms.append(shm_in)
             self.detection_shms.append(shm_out)
 


Test patch: