Repository: blakeblackshear/frigate

Issue:
Bug: wrong camera dimension calculation for birdseye
### Describe the problem you are having

```
2023-06-14 21:37:08.347713182  Traceback (most recent call last):
2023-06-14 21:37:08.347717528    File "/opt/frigate/frigate/output.py", line 549, in update
2023-06-14 21:37:08.347718362      updated_frame = self.update_frame()
2023-06-14 21:37:08.347719160    File "/opt/frigate/frigate/output.py", line 525, in update_frame
2023-06-14 21:37:08.347719827      self.copy_to_position(
2023-06-14 21:37:08.347720573    File "/opt/frigate/frigate/output.py", line 251, in copy_to_position
2023-06-14 21:37:08.347721350      copy_yuv_to_position(
2023-06-14 21:37:08.347722078    File "/opt/frigate/frigate/util.py", line 481, in copy_yuv_to_position
2023-06-14 21:37:08.347722667      destination_frame[
2023-06-14 21:37:08.347723383  ValueError: could not broadcast input array from shape (1080,1920) into shape (1080,1280)

```

this bug introduced in PR #6734 
if I comment out this lines:
```
 if camera_aspect < canvas_aspect:
                    camera_dims[0] *= camera_aspect / canvas_aspect
                    camera_dims[1] *= camera_aspect / canvas_aspect
``` 
always will work again



### Version

 dev-ff1b5c8

### Frigate config file

```yaml
detectors:
  deepstack:      
    type: deepstack
    api_url: http://192.168.88.50:5500/v1/vision/custom/ipcam-yolov5x6
    api_timeout: 0.5

go2rtc:
  ffmpeg:
    h264_nvenc: "-codec:v h264_nvenc -preset:v medium -b:v 4096k -maxrate 4096k -bufsize 4096k -r 24 -pix_fmt yuv420p -g 48"

  rtsp:
    default_query: "video&audio"

  webrtc:
    listen: ":8555"
    candidates:
      - 192.168.88.60:8555

  streams:
    kitchen: "rtsp://192.168.88.18:8080/video/h264#audio=aac#audio=pcm"
    livingroom: "rtsp://192.168.88.178:8080/video/h264#audio=aac"
    yard: rtsp://192.168.88.50:8554/yard
    secondfloor: rtsp://192.168.88.50:8554/secondfloor
    welcome: rtsp://192.168.88.50:8554/welcome

birdseye:
  enabled: True
  mode: continuous
  restream: True

ffmpeg:
  global_args: -hide_banner -loglevel debug
  hwaccel_args: preset-nvidia-h264
  #input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1
  input_args: preset-rtsp-restream
  output_args:
    record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v hevc_nvenc -tag:v hvc1 -c:a copy

record:
  enabled: True
  expire_interval: 60
  retain:
    days: 10
    mode: motion
  events:
    pre_capture: 30
    post_capture: 120
    retain:
      default: 7
      mode: motion
      objects:
        person: 180
        bottle: 1
        car: 7

snapshots:
  enabled: True
  height: 1080

cameras:
  livingroom:
    ffmpeg:
      inputs:
        - path: rtsp://localhost:8554/livingroom
          roles:
            - detect
            - record
    mqtt:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 1080
      quality: 100
    detect:
      fps: 10
      width: 1920
      height: 1080
    motion:
      mask:
        - 1763,0,1920,368,1268,506,924,530,789,509,695,509,686,537,680,633,659,899,0,886,0,0

  kitchen:
    ffmpeg:
      inputs:
   
        - path: rtsp://localhost:8554/kitchen
          roles:
            - detect
            - record
     
    mqtt:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 960
      quality: 100
    detect:
      fps: 10
      width: 1280
      height: 960
  yard:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.88.50:8554/yard
          roles:
            - detect
            - record
             
    detect:
      width: 1920
      height: 1080
      fps: 5
    mqtt:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 1080
      quality: 100
       
    motion:
      mask:
        - 0,0,0,720,273,720,262,535,410,518,647,477,662,236,966,253,972,0
        - 1280,0,1280,692,1103,607,1058,602,1054,720,943,720,965,0
    zones:
      gates:
        coordinates: 783,278,775,337,694,327,696,269
    record:
      events:
        pre_capture: 30
        post_capture: 120
        retain:
          default: 7
          mode: motion

  secondfloor:
    ffmpeg:
      inputs:
       
        - path: rtsp://192.168.88.50:8554/secondfloor
          roles:
            - detect
            - record
     
      hwaccel_args: preset-nvidia-h264
    detect:
      width: 1920
      height: 1080
      fps: 10
    mqtt:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 1080
      quality: 100

  welcome:
    ffmpeg:
      inputs:
       
        - path: rtsp://192.168.88.50:8554/welcome
          roles:
            - detect
            - record
      hwaccel_args: preset-nvidia-h264
     
    detect:
      width: 1920
      height: 1080
      fps: 10
    mqtt:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      quality: 100
      height: 1080
  terasse:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.88.50:8554/terasse?mp4 #rtsp://192.168.88.50:8554/terasse?mp4
          roles:
            - detect
            - record
              #- path: "rtsp://192.168.88.195:554/user=admin_password=QuuOEfOI_channel=0_stream=0.sdp?real_stream#hardware=cuda"
              #roles:
              #- restream
      hwaccel_args: preset-nvidia-h264
      
    mqtt:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 1288
      quality: 100
    detect:
      width: 2288
      height: 1288
      fps: 10
```


### Relevant log output

```shell
2023-06-14 21:37:08.347713182  Traceback (most recent call last):
2023-06-14 21:37:08.347717528    File "/opt/frigate/frigate/output.py", line 549, in update
2023-06-14 21:37:08.347718362      updated_frame = self.update_frame()
2023-06-14 21:37:08.347719160    File "/opt/frigate/frigate/output.py", line 525, in update_frame
2023-06-14 21:37:08.347719827      self.copy_to_position(
2023-06-14 21:37:08.347720573    File "/opt/frigate/frigate/output.py", line 251, in copy_to_position
2023-06-14 21:37:08.347721350      copy_yuv_to_position(
2023-06-14 21:37:08.347722078    File "/opt/frigate/frigate/util.py", line 481, in copy_yuv_to_position
2023-06-14 21:37:08.347722667      destination_frame[
2023-06-14 21:37:08.347723383  ValueError: could not broadcast input array from shape (1080,1920) into shape (1080,1280)
```


### Frigate stats

_No response_

### Operating system

UNRAID

### Install method

Docker CLI

### Coral version

USB

### Any other information that may be helpful

_No response_
Birdseye autolayout is less optimized than previous grid layout
### Describe the problem you are having

*image removed due to privacy*


I circled the boundaries of the cameras with red frames for clarity

cameras dimensions, from left top: 1280x960, 1920x1080, 1920x1080, 2288x1288 (no mistake here), 1920x1080, 1920x1080

### Version

dev-3efa77f

### Frigate config file

```yaml
see #6802
```


### Relevant log output

```shell
no relevant
```


### FFprobe output from your camera

```shell
no relevant
```


### Frigate stats

_No response_

### Operating system

UNRAID

### Install method

Docker CLI

### Coral version

USB

### Network connection

Wired

### Camera make and model

n/a

### Any other information that may be helpful

_No response_


Discussion:

Can you elaborate what you mean by broken? This looks to be working as expected unless I'm misunderstanding something 

Resolving patch:
diff --git a/frigate/output.py b/frigate/output.py
--- a/frigate/output.py
+++ b/frigate/output.py
@@ -1,6 +1,7 @@
 import datetime
 import glob
 import logging
+import math
 import multiprocessing as mp
 import os
 import queue
@@ -269,161 +270,118 @@ def camera_active(self, mode, object_box_count, motion_box_count):
     def update_frame(self):
         """Update to a new frame for birdseye."""
 
-        def calculate_two_cam_layout(canvas, cameras_to_add: list[str]) -> tuple[any]:
-            """Calculate the optimal layout for 2 cameras."""
-            first_camera = cameras_to_add[0]
-            first_camera_dims = self.cameras[first_camera]["dimensions"].copy()
-            second_camera = cameras_to_add[1]
-            second_camera_dims = self.cameras[second_camera]["dimensions"].copy()
-
-            # check for optimal layout
-            if first_camera_dims[0] + second_camera_dims[0] < canvas_width:
-                # place cameras horizontally
-                first_scaled_width = int(
-                    canvas_height * first_camera_dims[0] / first_camera_dims[1]
-                )
-                second_scaled_width = int(
-                    canvas_height * second_camera_dims[0] / second_camera_dims[1]
-                )
-                first_height = canvas_height
-                second_height = canvas_height
-
-                if first_scaled_width + second_scaled_width > canvas_width:
-                    if first_scaled_width > second_scaled_width:
-                        first_scaled_width = canvas_width - second_scaled_width
-                        first_height = int(
-                            first_scaled_width
-                            * first_camera_dims[1]
-                            / first_camera_dims[0]
-                        )
-                    else:
-                        second_scaled_width = canvas_width - first_scaled_width
-                        second_height = int(
-                            second_scaled_width
-                            * second_camera_dims[1]
-                            / second_camera_dims[0]
-                        )
-
-                return [
-                    [
-                        (
-                            first_camera,
-                            (0, 0, first_scaled_width, first_height),
-                        ),
-                        (
-                            second_camera,
-                            (
-                                first_scaled_width + 1,
-                                0,
-                                second_scaled_width,
-                                second_height,
-                            ),
-                        ),
-                    ],
-                ]
-            else:
-                # place cameras vertically
-                top_scaled_width = int(
-                    (canvas_height / 2) * first_camera_dims[0] / first_camera_dims[1]
-                )
-                bottom_scaled_width = int(
-                    (canvas_height / 2) * second_camera_dims[0] / second_camera_dims[1]
-                )
-                return [
-                    [
-                        (
-                            first_camera,
-                            (0, 0, top_scaled_width, int(canvas_height / 2)),
-                        )
-                    ],
-                    [
-                        (
-                            second_camera,
-                            (
-                                0,
-                                int(canvas_height / 2),
-                                bottom_scaled_width,
-                                int(canvas_height / 2),
-                            ),
-                        )
-                    ],
-                ]
-
         def calculate_layout(
             canvas, cameras_to_add: list[str], coefficient
         ) -> tuple[any]:
-            """Calculate the optimal layout for 3+ cameras."""
+            """Calculate the optimal layout for 2+ cameras."""
             camera_layout: list[list[any]] = []
             camera_layout.append([])
-            canvas_aspect = canvas[0] / canvas[1]
+            canvas_gcd = math.gcd(canvas[0], canvas[1])
+            canvas_aspect_x = (canvas[0] / canvas_gcd) * coefficient
+            canvas_aspect_y = (canvas[0] / canvas_gcd) * coefficient
             starting_x = 0
             x = starting_x
             y = 0
             y_i = 0
-            max_height = 0
+            max_y = 0
             for camera in cameras_to_add:
                 camera_dims = self.cameras[camera]["dimensions"].copy()
-                camera_aspect = camera_dims[0] / camera_dims[1]
+                camera_gcd = math.gcd(camera_dims[0], camera_dims[1])
+                camera_aspect_x = camera_dims[0] / camera_gcd
+                camera_aspect_y = camera_dims[1] / camera_gcd
+
+                if round(camera_aspect_x / camera_aspect_y, 1) == 1.8:
+                    # account for slightly off 16:9 cameras
+                    camera_aspect_x = 16
+                    camera_aspect_y = 9
+                elif round(camera_aspect_x / camera_aspect_y, 1) == 1.3:
+                    # make 4:3 cameras the same relative size as 16:9
+                    camera_aspect_x = 12
+                    camera_aspect_y = 9
 
                 if camera_dims[1] > camera_dims[0]:
                     portrait = True
-                elif camera_aspect < canvas_aspect:
-                    # if the camera aspect ratio is less than canvas aspect ratio, it needs to be scaled down to fit
-                    camera_dims[0] *= camera_aspect / canvas_aspect
-                    camera_dims[1] *= camera_aspect / canvas_aspect
-                    portrait = False
                 else:
                     portrait = False
 
-                if (x + camera_dims[0] * coefficient) <= canvas[0]:
+                if (x + camera_aspect_x) <= canvas_aspect_x:
                     # insert if camera can fit on current row
-                    scaled_width = int(camera_dims[0] * coefficient)
                     camera_layout[y_i].append(
                         (
                             camera,
                             (
-                                x,
-                                y,
-                                scaled_width,
-                                int(camera_dims[1] * coefficient),
+                                camera_aspect_x,
+                                camera_aspect_y,
                             ),
                         )
                     )
-                    x += scaled_width
 
                     if portrait:
-                        starting_x = scaled_width
+                        starting_x = camera_aspect_x
                     else:
-                        max_height = max(
-                            max_height,
-                            int(camera_dims[1] * coefficient),
+                        max_y = max(
+                            max_y,
+                            camera_aspect_y,
                         )
+
+                    x += camera_aspect_x
                 else:
                     # move on to the next row and insert
-                    y += max_height
+                    y += max_y
                     y_i += 1
                     camera_layout.append([])
                     x = starting_x
 
-                    if camera_dims[0] * coefficient > canvas_width:
-                        safe_coefficient = 1
-                    else:
-                        safe_coefficient = coefficient
+                    if x + camera_aspect_x > canvas_aspect_x:
+                        return None
 
                     camera_layout[y_i].append(
                         (
                             camera,
-                            (
-                                x,
-                                y,
-                                int(camera_dims[0] * safe_coefficient),
-                                int(camera_dims[1] * safe_coefficient),
-                            ),
+                            (camera_aspect_x, camera_aspect_y),
                         )
                     )
-                    x += int(camera_dims[0] * safe_coefficient)
+                    x += camera_aspect_x
+
+            if y + max_y > canvas_aspect_y:
+                return None
+
+            row_height = int(canvas_height / coefficient)
+
+            final_camera_layout = []
+            starting_x = 0
+            y = 0
+
+            for row in camera_layout:
+                final_row = []
+                x = starting_x
+                for cameras in row:
+                    camera_dims = self.cameras[cameras[0]]["dimensions"].copy()
 
-            return (camera_layout, y + max_height)
+                    if camera_dims[1] > camera_dims[0]:
+                        scaled_height = int(row_height * coefficient)
+                        scaled_width = int(
+                            scaled_height * camera_dims[0] / camera_dims[1]
+                        )
+                        starting_x = scaled_width
+                    else:
+                        scaled_height = row_height
+                        scaled_width = int(
+                            scaled_height * camera_dims[0] / camera_dims[1]
+                        )
+
+                    if (
+                        x + scaled_width > canvas_width
+                        or y + scaled_height > canvas_height
+                    ):
+                        return None
+
+                    final_row.append((cameras[0], (x, y, scaled_width, scaled_height)))
+                    x += scaled_width
+                y += row_height
+                final_camera_layout.append(final_row)
+
+            return final_camera_layout
 
         # determine how many cameras are tracking objects within the last 30 seconds
         active_cameras = set(
@@ -493,30 +451,28 @@ def calculate_layout(
                         )
                     ]
                 ]
-            elif len(active_cameras) == 2:
-                self.camera_layout = calculate_two_cam_layout(
-                    (canvas_width, canvas_height), active_cameras_to_add
-                )
             else:
                 # calculate optimal layout
-                coefficient = 1.0
+                coefficient = 2
                 calculating = True
 
                 # decrease scaling coefficient until height of all cameras can fit into the birdseye canvas
                 while calculating:
-                    layout_candidate, total_height = calculate_layout(
+                    layout_candidate = calculate_layout(
                         (canvas_width, canvas_height),
                         active_cameras_to_add,
                         coefficient,
                     )
 
-                    if (canvas_height * 0.75) < total_height <= canvas_height:
-                        calculating = False
-                    elif total_height < canvas_height * 0.75:
-                        coefficient += 0.1
-                        calculating = False
-                    else:
-                        coefficient -= 0.1
+                    if not layout_candidate:
+                        if coefficient < 10:
+                            coefficient += 1
+                            continue
+                        else:
+                            logger.error("Error finding appropriate birdseye layout")
+                            return
+
+                    calculating = False
 
                 self.camera_layout = layout_candidate
 


Test patch: