Repository: blakeblackshear/frigate

Issue:
[BUG] Connection already opened
**Describe the bug**
A clear and concise description of what your issue is.
After a while / restart or Home-Assistant I get a connection error. Does this mean it did not close the connection when restarting HASS?
**Version of frigate**
1.13
**Config file**
Include your full config file wrapped in triple back ticks.
```yaml
not related```

**Frigate container logs**
```
https://pastebin.ddns.me/hoviyalasa.rb```

**Frigate stats**
```json
Internal Server Error```

**FFprobe from your camera**

Run the following command and paste output below
```
ffprobe <stream_url>
```

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Computer Hardware**
 - OS:  Ubuntu, 
 - Install method: Addon

 - Coral Version: PCIe
 - Network Setup: WiFi]






Discussion:
Same problem for me with Frigate 0.8.4-5043040

```
2021-03-19T21:44:03Z {'REMOTE_ADDR': '127.0.0.1', 'REMOTE_PORT': '39814', 'HTTP_HOST': 'frigate', (hidden keys: 23)} failed with OperationalError

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/flask_sockets.py", line 40, in __call__
    handler, values = adapter.match()
  File "/usr/local/lib/python3.8/dist-packages/werkzeug/routing.py", line 1945, in match
    raise NotFound()
werkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/gevent/pywsgi.py", line 999, in handle_one_response
    self.run_application()
  File "/usr/local/lib/python3.8/dist-packages/geventwebsocket/handler.py", line 87, in run_application
    return super(WebSocketHandler, self).run_application()
  File "/usr/local/lib/python3.8/dist-packages/gevent/pywsgi.py", line 945, in run_application
    self.result = self.application(self.environ, self.start_response)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/flask_sockets.py", line 48, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1948, in full_dispatch_request
    rv = self.preprocess_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2242, in preprocess_request
    rv = func()
  File "/opt/frigate/frigate/http.py", line 89, in _db_connect
    database.connect()
  File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 3061, in connect
    raise OperationalError('Connection already opened.')
peewee.OperationalError: Connection already opened.
```

The issue seems to arise randomly after some time.
When the issue occurs, the API returns HTTP 500.

MQTT status topic stays on "online". Could be a good improvement to catch the exception and set the topic to offline (or failure?)
A container restart solves the issue. So as a workaround it makes sense to exit for this kind of issue, then the container will be restarted by Docker (considered `restart` is not `no`) and the issue solved. What is your opinion on that?
Seeing same here as well. Running docker - blakeblackshear/frigate:0.8.4-amd64
Restart does the trick, but it seems to pop up again within a few hours. 
Seems that this is the same as: https://github.com/blakeblackshear/frigate/discussions/853

I also started experiencing this, frigate stops working about once a day 
I removed double take and no errors for 24 hours.well. must be related to either frigate or double take .

Another thing ..if frigate would support face recognition I wouldn't be needed to use double take ..well... We will see 
I removed my automation that requests an the image via ha integration API and it slowed down the frequency of the crashes. I'll try and expose the images another way maybe. It It was recommended on #853 to downgrade, but I believe you'll have to downgrade both frigate and the ha add-on, which I didn't really want to do. Does anyone have a good method to catch the this particular crash and reboot via docker?
getting this too on docker amd64 stable. Tried downgrading to 0.8.1 but my config isn't valid:
`Error parsing config: extra keys not allowed @ data['cameras']['front']['ffmpeg']['input_args']`
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Resolving patch:
diff --git a/frigate/app.py b/frigate/app.py
--- a/frigate/app.py
+++ b/frigate/app.py
@@ -2,26 +2,25 @@
 import logging
 import multiprocessing as mp
 import os
+import signal
+import sys
+import threading
 from logging.handlers import QueueHandler
 from typing import Dict, List
-import sys
-import signal
 
 import yaml
-from gevent import pywsgi
-from geventwebsocket.handler import WebSocketHandler
 from peewee_migrate import Router
 from playhouse.sqlite_ext import SqliteExtDatabase
 from playhouse.sqliteq import SqliteQueueDatabase
 
 from frigate.config import FrigateConfig
-from frigate.const import RECORD_DIR, CLIPS_DIR, CACHE_DIR
+from frigate.const import CACHE_DIR, CLIPS_DIR, RECORD_DIR
 from frigate.edgetpu import EdgeTPUProcess
-from frigate.events import EventProcessor, EventCleanup
+from frigate.events import EventCleanup, EventProcessor
 from frigate.http import create_app
 from frigate.log import log_process, root_configurer
 from frigate.models import Event, Recordings
-from frigate.mqtt import create_mqtt_client
+from frigate.mqtt import create_mqtt_client, MqttSocketRelay
 from frigate.object_processing import TrackedObjectProcessor
 from frigate.output import output_frames
 from frigate.record import RecordingMaintainer
@@ -121,8 +120,8 @@ def set_log_levels(self):
         for log, level in self.config.logger.logs.items():
             logging.getLogger(log).setLevel(level)
 
-        if not "geventwebsocket.handler" in self.config.logger.logs:
-            logging.getLogger("geventwebsocket.handler").setLevel("ERROR")
+        if not "werkzeug" in self.config.logger.logs:
+            logging.getLogger("werkzeug").setLevel("ERROR")
 
     def init_queues(self):
         # Queues for clip processing
@@ -166,12 +165,17 @@ def init_web_server(self):
             self.db,
             self.stats_tracking,
             self.detected_frames_processor,
-            self.mqtt_client,
         )
 
     def init_mqtt(self):
         self.mqtt_client = create_mqtt_client(self.config, self.camera_metrics)
 
+    def start_mqtt_relay(self):
+        self.mqtt_relay = MqttSocketRelay(
+            self.mqtt_client, self.config.mqtt.topic_prefix
+        )
+        self.mqtt_relay.start()
+
     def start_detectors(self):
         model_shape = (self.config.model.height, self.config.model.width)
         for name in self.config.cameras.keys():
@@ -267,10 +271,6 @@ def start_camera_capture_processes(self):
             capture_process.start()
             logger.info(f"Capture process started for {name}: {capture_process.pid}")
 
-    def start_birdseye_outputter(self):
-        self.birdseye_outputter = BirdsEyeFrameOutputter(self.stop_event)
-        self.birdseye_outputter.start()
-
     def start_event_processor(self):
         self.event_processor = EventProcessor(
             self.config,
@@ -330,6 +330,7 @@ def start(self):
         self.start_camera_capture_processes()
         self.init_stats()
         self.init_web_server()
+        self.start_mqtt_relay()
         self.start_event_processor()
         self.start_event_cleanup()
         self.start_recording_maintainer()
@@ -343,12 +344,8 @@ def receiveSignal(signalNumber, frame):
 
         signal.signal(signal.SIGTERM, receiveSignal)
 
-        server = pywsgi.WSGIServer(
-            ("127.0.0.1", 5001), self.flask_app, handler_class=WebSocketHandler
-        )
-
         try:
-            server.serve_forever()
+            self.flask_app.run(host="127.0.0.1", port=5001, debug=False)
         except KeyboardInterrupt:
             pass
 
@@ -358,6 +355,7 @@ def stop(self):
         logger.info(f"Stopping...")
         self.stop_event.set()
 
+        self.mqtt_relay.stop()
         self.detected_frames_processor.join()
         self.event_processor.join()
         self.event_cleanup.join()
diff --git a/frigate/http.py b/frigate/http.py
--- a/frigate/http.py
+++ b/frigate/http.py
@@ -11,7 +11,7 @@
 from pathlib import Path
 
 import cv2
-import gevent
+
 import numpy as np
 from flask import (
     Blueprint,
@@ -22,7 +22,7 @@
     make_response,
     request,
 )
-from flask_sockets import Sockets
+
 from peewee import SqliteDatabase, operator, fn, DoesNotExist, Value
 from playhouse.shortcuts import model_to_dict
 
@@ -35,74 +35,6 @@
 logger = logging.getLogger(__name__)
 
 bp = Blueprint("frigate", __name__)
-ws = Blueprint("ws", __name__)
-
-
-class MqttBackend:
-    """Interface for registering and updating WebSocket clients."""
-
-    def __init__(self, mqtt_client, topic_prefix):
-        self.clients = list()
-        self.mqtt_client = mqtt_client
-        self.topic_prefix = topic_prefix
-
-    def register(self, client):
-        """Register a WebSocket connection for Mqtt updates."""
-        self.clients.append(client)
-
-    def publish(self, message):
-        try:
-            json_message = json.loads(message)
-            json_message = {
-                "topic": f"{self.topic_prefix}/{json_message['topic']}",
-                "payload": json_message["payload"],
-                "retain": json_message.get("retain", False),
-            }
-        except:
-            logger.warning("Unable to parse websocket message as valid json.")
-            return
-
-        logger.debug(
-            f"Publishing mqtt message from websockets at {json_message['topic']}."
-        )
-        self.mqtt_client.publish(
-            json_message["topic"],
-            json_message["payload"],
-            retain=json_message["retain"],
-        )
-
-    def run(self):
-        def send(client, userdata, message):
-            """Sends mqtt messages to clients."""
-            try:
-                logger.debug(f"Received mqtt message on {message.topic}.")
-                ws_message = json.dumps(
-                    {
-                        "topic": message.topic.replace(f"{self.topic_prefix}/", ""),
-                        "payload": message.payload.decode(),
-                    }
-                )
-            except:
-                # if the payload can't be decoded don't relay to clients
-                logger.debug(
-                    f"MQTT payload for {message.topic} wasn't text. Skipping..."
-                )
-                return
-
-            for client in self.clients:
-                try:
-                    client.send(ws_message)
-                except:
-                    logger.debug(
-                        "Removing websocket client due to a closed connection."
-                    )
-                    self.clients.remove(client)
-
-        self.mqtt_client.message_callback_add(f"{self.topic_prefix}/#", send)
-
-    def start(self):
-        """Maintains mqtt subscription in the background."""
-        gevent.spawn(self.run)
 
 
 def create_app(
@@ -110,10 +42,8 @@ def create_app(
     database: SqliteDatabase,
     stats_tracking,
     detected_frames_processor,
-    mqtt_client,
 ):
     app = Flask(__name__)
-    sockets = Sockets(app)
 
     @app.before_request
     def _db_connect():
@@ -129,10 +59,6 @@ def _db_close(exc):
     app.detected_frames_processor = detected_frames_processor
 
     app.register_blueprint(bp)
-    sockets.register_blueprint(ws)
-
-    app.mqtt_backend = MqttBackend(mqtt_client, frigate_config.mqtt.topic_prefix)
-    app.mqtt_backend.start()
 
     return app
 
@@ -613,7 +539,7 @@ def vod(year_month, day, hour, camera):
 def imagestream(detected_frames_processor, camera_name, fps, height, draw_options):
     while True:
         # max out at specified FPS
-        gevent.sleep(1 / fps)
+        time.sleep(1 / fps)
         frame = detected_frames_processor.get_current_frame(camera_name, draw_options)
         if frame is None:
             frame = np.zeros((height, int(height * 16 / 9), 3), np.uint8)
@@ -626,16 +552,3 @@ def imagestream(detected_frames_processor, camera_name, fps, height, draw_option
             b"--frame\r\n"
             b"Content-Type: image/jpeg\r\n\r\n" + jpg.tobytes() + b"\r\n\r\n"
         )
-
-
-@ws.route("/ws")
-def echo_socket(socket):
-    current_app.mqtt_backend.register(socket)
-
-    while not socket.closed:
-        # Sleep to prevent *constant* context-switches.
-        gevent.sleep(0.1)
-
-        message = socket.receive()
-        if message:
-            current_app.mqtt_backend.publish(message)
diff --git a/frigate/mqtt.py b/frigate/mqtt.py
--- a/frigate/mqtt.py
+++ b/frigate/mqtt.py
@@ -1,7 +1,16 @@
+import json
 import logging
 import threading
+from wsgiref.simple_server import make_server
 
 import paho.mqtt.client as mqtt
+from ws4py.server.wsgirefserver import (
+    WebSocketWSGIHandler,
+    WebSocketWSGIRequestHandler,
+    WSGIServer,
+)
+from ws4py.server.wsgiutils import WebSocketWSGIApplication
+from ws4py.websocket import WebSocket
 
 from frigate.config import FrigateConfig
 
@@ -117,8 +126,15 @@ def on_connect(client, userdata, flags, rc):
         )
 
     if not mqtt_config.tls_ca_certs is None:
-        if not mqtt_config.tls_client_cert is None and not mqtt_config.tls_client_key is None:
-            client.tls_set(mqtt_config.tls_ca_certs, mqtt_config.tls_client_cert, mqtt_config.tls_client_key)
+        if (
+            not mqtt_config.tls_client_cert is None
+            and not mqtt_config.tls_client_key is None
+        ):
+            client.tls_set(
+                mqtt_config.tls_ca_certs,
+                mqtt_config.tls_client_cert,
+                mqtt_config.tls_client_key,
+            )
         else:
             client.tls_set(mqtt_config.tls_ca_certs)
     if not mqtt_config.tls_insecure is None:
@@ -151,3 +167,79 @@ def on_connect(client, userdata, flags, rc):
         )
 
     return client
+
+
+class MqttSocketRelay:
+    def __init__(self, mqtt_client, topic_prefix):
+        self.mqtt_client = mqtt_client
+        self.topic_prefix = topic_prefix
+
+    def start(self):
+        class MqttWebSocket(WebSocket):
+            topic_prefix = self.topic_prefix
+            mqtt_client = self.mqtt_client
+
+            def received_message(self, message):
+                try:
+                    json_message = json.loads(message.data.decode("utf-8"))
+                    json_message = {
+                        "topic": f"{self.topic_prefix}/{json_message['topic']}",
+                        "payload": json_message["payload"],
+                        "retain": json_message.get("retain", False),
+                    }
+                except Exception as e:
+                    logger.warning("Unable to parse websocket message as valid json.")
+                    return
+
+                logger.debug(
+                    f"Publishing mqtt message from websockets at {json_message['topic']}."
+                )
+                self.mqtt_client.publish(
+                    json_message["topic"],
+                    json_message["payload"],
+                    retain=json_message["retain"],
+                )
+
+        # start a websocket server on 5002
+        WebSocketWSGIHandler.http_version = "1.1"
+        self.websocket_server = make_server(
+            "127.0.0.1",
+            5002,
+            server_class=WSGIServer,
+            handler_class=WebSocketWSGIRequestHandler,
+            app=WebSocketWSGIApplication(handler_cls=MqttWebSocket),
+        )
+        self.websocket_server.initialize_websockets_manager()
+        self.websocket_thread = threading.Thread(
+            target=self.websocket_server.serve_forever
+        )
+
+        def send(client, userdata, message):
+            """Sends mqtt messages to clients."""
+            try:
+                logger.debug(f"Received mqtt message on {message.topic}.")
+                ws_message = json.dumps(
+                    {
+                        "topic": message.topic.replace(f"{self.topic_prefix}/", ""),
+                        "payload": message.payload.decode(),
+                    }
+                )
+            except Exception as e:
+                # if the payload can't be decoded don't relay to clients
+                logger.debug(
+                    f"MQTT payload for {message.topic} wasn't text. Skipping..."
+                )
+                return
+
+            self.websocket_server.manager.broadcast(ws_message)
+
+        self.mqtt_client.message_callback_add(f"{self.topic_prefix}/#", send)
+
+        self.websocket_thread.start()
+
+    def stop(self):
+        self.websocket_server.manager.close_all()
+        self.websocket_server.manager.stop()
+        self.websocket_server.manager.join()
+        self.websocket_server.shutdown()
+        self.websocket_thread.join()


Test patch: