Video surveillance system with vantage point transformation

A method for monitoring a surveillance area includes receiving a plurality of video streams that are captured by a plurality of video cameras in the surveillance area, each of the plurality of video streams capturing one or more objects in the surveillance area that are within a camera field of view (FOV) of the corresponding video camera and from a camera vantage point of the corresponding video camera. A vantage point transformation is applied to each of two or more objects captured in two or more of the plurality of video streams, the vantage point transformation transforming a view of each of the two or more objects from the camera vantage point of the corresponding video camera to a common vantage point. The transformed view of each of the two or more objects from the common vantage point is rendered on a display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority pursuant to 35 U.S.C. 119(a) to India Patent Application No. 202111020181, filed May 3, 2021, which application is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates generally to video surveillance systems. More particularly, the present disclosure relates to video surveillance systems that enable vantage point transformations.

BACKGROUND

A number of video surveillance systems employ video cameras that are installed or otherwise arranged around a surveillance area such as a city, a portion of a city, a facility or a building. Video surveillance systems may also include mobile video cameras, such as drones carrying video cameras. Each video camera has a vantage point that corresponds to its physical location and a field of view that corresponds to what can be seen by that particular video camera at its particular physical location. In the case of a mobile video camera, it will be appreciated that its vantage point can change. What would be desirable is a way to combine video images from a plurality of the video cameras and perform a vantage point transformation to provide a “birds eye” view of the surveillance area that is controllable by an operator.

SUMMARY

The present disclosure relates to video surveillance systems. In an example, a method for monitoring a surveillance area is provided. The method includes receiving a plurality of video streams that are captured by a plurality of video cameras in the surveillance area, wherein each of the plurality of video cameras has a camera vantage point and a camera field of view (FOV). Each of the plurality of video streams captures one or more objects in the surveillance area that are within the camera FOV of the corresponding video camera and from the camera vantage point of the corresponding video camera. A vantage point transformation is applied to each of two or more objects captured in two or more of the plurality of video streams. The vantage point transformation transforming a view of each of the two or more objects from the camera vantage point of the corresponding video camera to a common vantage point. The transformed view of each of the two or more objects from the common vantage point is then rendered on a display. In some cases, video streams from two or more video cameras are stitched together to produce the view of one or more of the objects.

In another example, performing a search may include receiving a search query from an operator. In response, a number of videos may be searched for objects that meet the search query. The location of matching objects may be displayed on a transformed view from a common vantage point at a first frame rate. Operator input may be received that zooms the common vantage point into a selected one of the matching objects. The selected matching object from the zoomed-in common vantage point may be displayed at a second frame rate that may or may not be higher than the first frame rate.

In another example, a method of operating a surveillance system that includes a plurality of video cameras is provided. Each of the video cameras is configured to provide video streams from a corresponding camera vantage point, and the surveillance system includes a surveillance system controller. The surveillance system controller may be centrally provided, such as in the cloud, or may be distributed such as among edge controllers at or near the various video cameras. The illustrative method includes the surveillance system controller analyzing video streams provided by the plurality of video cameras in order to find a common object in a first video frame from a first video stream and a second video frame from a second video stream. The common object is tagged with an object identifier. View information for the tagged common object that includes view information from the camera vantage point of each of the first video stream and the second video stream is stored. A vantage point transformation is applied to the common object, the vantage point transformation transforming the view information for the common object from the camera vantage points of the first video stream and the second video stream to a common vantage point. The transformed view information of the common object from the common vantage point is rendered on a display. User input may be received to move the common vantage point to an updated common vantage point, and once moved, the vantage point transformation is applied using the updated common vantage point and the transformed view information of the common object is rendered from the updated common vantage point.

In another example, a drone is configured for use in a surveillance system. The drone includes a video camera having a camera Field of View (FOV), a memory, a transceiver and a controller that is operably coupled to the video camera, the memory and the transceiver. The controller is configured to capture a first frame of a video of an incident at a first location using the video camera, and to determine a flight path to follow the incident. The controller is configured to fly the drone along the flight path and to capture a second frame of the video of the incident at a second location using the video camera, wherein the second location is determined such that that there is an overlap in the camera Field of View (FOV) between the first frame of the video and the second frame of the video. The controller is configured to transmit the resulting video via the transceiver.

In another example, a surveillance system is configured to provide surveillance of a surveillance area. The surveillance system includes a plurality of video cameras disposed within the surveillance area. Each of the plurality of video cameras is configured to capture and store a video stream corresponding to a field of view of the particular video camera. A surveillance system monitoring controller is operably coupled with each of the plurality of video cameras via a high speed wireless network. The surveillance system monitoring controller includes a high speed input configured to receive video streams from one or more of the plurality of video cameras via the high speed wireless network, a memory operably coupled to the high speed input and configured to store the received video streams, and a controller that is operably coupled to the high speed input and to the memory. The controller is configured to analyze each of the video streams in order to find a first common landmark in both a first video frame from a first video stream and a second video frame from a second video stream. When the first common landmark is present in both the first video frame from the first video stream and the second video frame from the second video stream, the controller is configured to stitch together the first video frame from the first video stream and the second video frame from the second video stream and place the stitched together image into a master image. When there is no first common landmark present in both the first video frame from the first video stream and the second video frame from the second video stream, the controller is configured to populate the master image with the first video frame from the first video stream and the second video frame from the second video stream with each of the first video frames disposed within the master image at relative locations corresponding to physical locations of the views included in the first video frame and the second video frame. The controller is configured to transform the master image into a top view of the master image using an image transformation.

In another example, a method of operating a surveillance system that includes a surveillance system controller and a plurality of video cameras that are configured to provided video streams is provided. The illustrative method includes the surveillance system controller analyzing video streams provided by the plurality of video cameras in order to find a common landmark in a first video frame from a first video stream and a second video frame from a second video stream. When the common landmark is present in both the first video frame from the first video stream and the second video frame from the second video stream, the surveillance system controller stitches together the first video frame from the first video stream and the second video frame from the second video stream and places the stitched together image into a master image. When there is no common landmark present in both the first video frame from the first video stream and the second video frame from the second video stream, the surveillance system controller populates the master image with the first video frame from the first video stream and the second video frame from the second video stream with each of the first video frames disposed within the master image at relative locations corresponding to physical locations of the views included in the first video frame and the second video frame. The surveillance system controller translates the master image into a top view of the master image.

In another example, a drone is configured for use in a surveillance system that includes a plurality of video cameras disposed within a surveillance area. The drone includes a video camera, a memory, a cellular transceiver, and a controller that is operably coupled to the video camera, the memory and the cellular transceiver. The controller is configured to receive instructions to fly to a particular location at which an incident is believed to be occurring and to capture a first video frame of the incident using the video camera. The controller is configured to fly to a second location away from the particular location to follow the incident, to determine a time to capture a second video frame of the incident such that there is sufficient overlap between the first video frame and the second video frame to be able to stitch together the first video fame and the second video frame and to capture a second video frame of the incident. The controller is configured to stitch together the first video frame and the second video frame to create a stitched video image and to transmit the stitched video image via the cellular transceiver.

DESCRIPTION

The following description should be read with reference to the drawings, in which like elements in different drawings are numbered in like fashion. The drawings, which are not necessarily to scale, depict examples that are not intended to limit the scope of the disclosure. Although examples are illustrated for the various elements, those skilled in the art will recognize that many of the examples provided have suitable alternatives that may be utilized.

FIG.1Ais a schematic block diagram of an illustrative surveillance system10that is configured to provide surveillance of a surveillance area. The illustrative surveillance system10includes a surveillance system controller12and a plurality of video cameras14that are disposed within the surveillance area. The video cameras14are individually labeled as14a,14b,14c. While a total of three video cameras are illustrated, it will be appreciated that the surveillance system10may include hundreds or even thousands of video cameras14that are disposed about a smart city, for example. At least some of the video cameras14may be fixed video cameras, meaning that they are each installed at a fixed location. At least some of the video cameras14may be mobile video cameras that are configured to move about within the surveillance area. For example, at least some of the video cameras14may be mounted within drones that are configured to fly around within the surveillance area, thereby providing video cameras at various locations and/or vertical positions within the surveillance area. In some cases, the mobile video cameras may be dash cameras of emergency vehicles, body cameras of emergency personnel such as police, and/or portable or wearable devices carried by citizens. These are just examples.

Each of the video cameras14include a vantage point16and a field of view (FOV)18. The vantage points16are individually labeled as16a,16b,16cand the FOVs18are individually labeled as18a,18b,18c. For each video camera14, its vantage point16may be defined at least in part upon where it is mounted, if permanently secured, or currently located, if it is mobile. For example, a particular video camera14may be mounted on the exterior of a building at the intersection of First Street and Main, at a height of ten feet. This mounting location may be considered as defining its vantage point16. As another example, a particular video camera14may be mounted at the intersection of Third Street and Sixteenth Avenue and at a height of twenty five feet such as on the exterior of a building or on a light post. This particular video camera14(mounted at a different location and a different height of twenty five feet) may be considered as having a vantage point16that is different from the vantage point16of the video camera that is mounted at a height of ten feet. A video camera14mounted to a drone will have yet a different vantage point16.

Some of the video cameras14may have a fixed FOV18that is dictated by where and how the cameras are installed, the lens installed on the camera, and so on. Some of the video cameras may have, for example, a 120 degree FOV or a 360 degree FOV. Some of the video cameras14may have a FOV18that is adjustable. For example, some of the video cameras14may be Pan, Tilt and Zoom (PTZ) cameras that can adjust their FOV by adjusting one or more of the Pan, the Tilt and the Zoom of the particular video cameras14.

The surveillance system controller12may be configured to control at least some aspects of operation of the surveillance system10. For example, the surveillance system controller12may be configured to provide instructions to at least some of the video cameras14to transmit video, for example, or to change one or more of the Pan, the Tilt and the Zoom of video cameras14that are PTZ cameras. The surveillance system controller12may be configured to control operation of any mobile video cameras that is part of the surveillance system10. The surveillance system controller12may be configured to carry out a number of different methods.FIGS.2through6are flow diagrams showing illustrative methods that may be orchestrated by the surveillance system controller12and thus carried out by the surveillance system10.

FIG.1Bis a schematic block diagram of an illustrative surveillance system11that is configured to provide surveillance of a surveillance area. The surveillance system11may include a plurality of video cameras15that are disposed within the surveillance area. The video cameras15are individually labeled as15a,15b,15c,15d. While a total of four video cameras15are illustrated, it will be appreciated that the surveillance system11may include hundreds or even thousands of video cameras15that are disposed about a smart city, for example. At least some of the video cameras15may be fixed video cameras, meaning that they are each installed at a fixed location. At least some of the video cameras15may be mobile video cameras that are configured to move about within the surveillance area. For example, at least some of the video cameras15may be mounted within drones that are configured to fly around within the surveillance area, thereby providing video cameras at various locations and/or vertical positions within the surveillance area. In some cases, the mobile video cameras may be dash cameras of emergency vehicles, body cameras of emergency personnel such as police, and/or portable or wearable devices carried by citizens. These are just examples.

Each of the video cameras15include a location17that may be given in terms of latitude and longitude and a field of view (FOV)18. The locations17are individually labeled as17a,17b,17c,17dand the FOVs18are individually labeled as18a,18b,18c,18d. For each video camera15, its location17may be defined at least in part upon where it is mounted, if permanently secured, or currently located, if it is mobile. Some of the video cameras15may have a fixed FOV18that is dictated by where and how the cameras are installed, the lens installed on the camera, and so on. Some of the video cameras15may have, for example, a 120 degree FOV or a 360 degree FOV. Some of the video cameras15may have a FOV18that is adjustable. For example, some of the video cameras15may be Pan, Tilt and Zoom (PTZ) cameras that can adjust their FOV by adjusting one or more of the Pan, the Tilt and the Zoom of the particular video cameras15.

The surveillance system11includes one or more edge devices19. Two edge devices19are shown, individually labeled as19aand19b. It will be appreciated that there may be a substantially greater number of edge devices19, for example. As shown, the video cameras15a,15band15care operably coupled with the edge device19aand the video camera15d(and possibly others) are operably coupled with the edge device19b. In some cases, the edge devices19may provide some of the functionality described with respect to the surveillance system controller12(FIG.1A). Some of the functionality described with respect to the surveillance system controller12may be provided by a cloud-based server13and/or a computer or workstation21that is operably coupled to the edge devices19via the cloud-based server13. The functionality of the surveillance system controller12may be centrally provided, such as via the cloud-based server13, or may be distributed between the edge devices19that can be located at or near the various video cameras15.

The cloud-based server13, the computer or workstation21and/or one or more of the edge devices19may be configured to control at least some aspects of operation of the surveillance system11. For example, the cloud-based server13, the computer or workstation21and/or one or more of the edge devices19may be configured to provide instructions to at least some of the video cameras15to transmit video, for example, or to change one or more of the Pan, the Tilt and the Zoom of video cameras15that are PTZ cameras. The cloud-based server13, the computer or workstation21and/or one or more of the edge devices19may be configured to control operation of any mobile video cameras that is part of the surveillance system11. The cloud-based server13, the computer or workstation21and/or one or more of the edge devices19may be configured to carry out a number of different methods.FIGS.2through6are flow diagrams showing illustrative methods that may be orchestrated by the surveillance system controller12and thus carried out by the surveillance system10. Alternatively, the illustrative methods shown inFIGS.2through6may also be orchestrated and thus carried out via the cloud-based server13, the computer or workstation21and/or one or more of the edge devices19.

FIG.2is a flow diagram showing an illustrative method20for monitoring a surveillance area. The method20includes receiving a plurality of video streams captured by a plurality of video cameras (such as the video cameras14) in the surveillance area, wherein each of the plurality of video cameras has a camera vantage point (such as the vantage point16) and a camera field of view (FOV) (such as the FOV18). Each of the plurality of video streams capturing one or more objects in the surveillance area that are within the camera FOV of the corresponding video camera and from the camera vantage point of the corresponding video camera, as indicated at block22. A vantage point transformation may be applied to each of two or more objects captured in two or more of the plurality of video streams, the vantage point transformation transforming a view of each of the two or more objects from the camera vantage point16of the corresponding video camera to a common vantage point, as indicated at block24. The common vantage point may correspond to a “birds eye” view of the surveillance area that can be moved and/or otherwise controlled by an operator of the surveillance system. The transformation may include a geometric transformation from each of the camera vantage points to the common vantage point.

In one example, say that the camera vantage point16of a first video camera14corresponds to a particular location and a height of ten feet and the camera vantage point16of a second video camera14corresponds to a position fifty yards south of the position of the first video camera14and a height of twenty five feet. The common vantage point may correspond to the camera vantage point of one of the video cameras14, or the common vantage point may correspond to a different position, such as a “birds eye” view of the surveillance area that can be moved and/or otherwise controlled by an operator of the surveillance system. Perhaps there is a desire for the common vantage point to correspond to a location intermediate the first video camera14and the second video camera14, and at a height intermediate the two. In another example, the common vantage point may correspond to a substantially greater vertical height, such as a top view or a desired birds eye view of the area.

The transformed view of each of the two or more objects from the common vantage point may be rendered on a display, as indicated at block26. In some instances, the method20may further include rendering on the display indicators that indicate regions of the surveillance area that are not within the camera FOV of any of the plurality of video cameras, as indicated at block28. The method20may, for example, additionally or alternatively include rendering on the display indicators that indicate regions of objects and/or regions of the surveillance area that are not within the camera FOV of any of the plurality of video cameras, as indicated at block30. This notifies an operator of areas of the surveillance area that are not covered by the video cameras of the surveillance system10.

FIG.3is a flow diagram showing an illustrative method32for monitoring a surveillance area. The method32includes receiving a plurality of video streams captured by a plurality of video cameras (such as the video cameras14) in the surveillance area, wherein each of the plurality of video cameras has a camera vantage point (such as the vantage point16) and a camera field of view (FOV) (such as the FOV18. Each of the plurality of video streams capturing one or more objects in the surveillance area that are within the camera FOV of the corresponding video camera and from the camera vantage point of the corresponding video camera, as indicated at block34. A vantage point transformation may be applied to each of two or more objects captured in two or more of the plurality of video streams, the vantage point transformation transforming a view of each of the two or more objects from the camera vantage point of the corresponding video camera to a common vantage point, as indicated at block36. In some cases, the view of each of the two or more objects are transformed and rendered from the common vantage point at a first frame rate. The transformed view of each of the two or more objects from the common vantage point may be rendered on a display, as indicated at block38.

In some instances, the method32includes receiving user input to move the common vantage point to an updated common vantage point, as indicated at block40. In some cases, moving the common vantage point may include one or more of panning, zooming, titling and rotating. As an example, it may be possible to zoom in the common vantage point to a zoomed in updated common vantage point. A vantage point transformation may be applied using the updated common vantage point, as indicated at block42. The transformed view of each of the two or more objects may be rendered from the updated common vantage point, as indicated at block44. In some cases, a view of the two or more objects from a zoomed in updated common vantage point may be rendered at a second frame rate that is higher than the first frame rate.

FIG.4is a flow diagram showing an illustrative method46for monitoring a surveillance area. The method46includes receiving a plurality of video streams captured by a plurality of video cameras (such as the video cameras14) in the surveillance area, wherein each of the plurality of video cameras has a camera vantage point (such as the vantage point16) and a camera field of view (FOV) (such as the FOV18). Each of the plurality of video streams capturing one or more objects in the surveillance area that are within the camera FOV of the corresponding video camera and from the camera vantage point of the corresponding video camera, as indicated at block48. A vantage point transformation may be applied to each of two or more objects captured in two or more of the plurality of video streams. The vantage point transformation transforming a view of each of the two or more objects from the camera vantage point of the corresponding video camera to a common vantage point, as indicated at block50. In some cases, the view of each of the two or more objects are transformed and rendered from the common vantage point at a first frame rate. The first frame rate may depend on the number of objects and the number of video cameras that are of interest. The fewer the objects and/or video cameras that are of interest (e.g. a zoomed in common vantage point and the smaller field of view may have fewer objects and fewer video cameras that are of interest), the higher the frame rate. The transformed view of each of the two or more objects from the common vantage point may be rendered on a display, as indicated at block52.

In some cases, the method46may further include identifying one or more objects in the surveillance area that are within the camera FOV of at least one of the plurality of video cameras, as indicated at block54. A determination is made as to when an object identified within the camera FOV of two or more of the plurality of video cameras corresponds to the same object, as indicated at block56. The object is tagged with an object identifier, as indicated at block58. The view information for the tagged object is stored that includes view information from the camera vantage point of each of the two or more of the plurality of video cameras that captured the tagged object over time, as indicated at block60. The view information may, for example, include view information captured from different cameras and at different times. In some cases, the vantage point transformation transforms the view information to the common vantage point.

FIG.5is a flow diagram showing an illustrative method62for monitoring a surveillance area. The method62includes receiving a plurality of video streams captured by a plurality of video cameras (such as the video cameras14) in the surveillance area, wherein each of the plurality of video cameras has a camera vantage point (such as the vantage point16) and a camera field of view (FOV) (such as the FOV18). Each of the plurality of video streams capturing one or more objects in the surveillance area that are within the camera FOV of the corresponding video camera and from the camera vantage point of the corresponding video camera, as indicated at block64. A vantage point transformation may be applied to each of two or more objects captured in two or more of the plurality of video streams, the vantage point transformation transforming a view of each of the two or more objects from the camera vantage point of the corresponding video camera to a common vantage point, as indicated at block66. In some cases, the view of each of the two or more objects are transformed and rendered from the common vantage point at a first frame rate. The first frame rate may depend on the number of objects and the number of video cameras that are of interest. The fewer the objects and/or video cameras that are of interest (e.g. a zoomed in common vantage point and the smaller field of view may have fewer objects and fewer video cameras that are of interest), the higher the frame rate. The transformed view of each of the two or more objects from the common vantage point may be rendered on a display, as indicated at block68.

In some cases, the method62further includes determining when the camera FOV of one of the plurality of video cameras overlaps with the camera FOV of another one of the plurality of video cameras, as indicated at block70. When the camera FOV of one of the plurality of video cameras overlaps with the camera FOV of another one of the plurality of video cameras, the corresponding video streams are stitched together, as indicated at block72.

FIG.6is a flow diagram showing an illustrative method74of operating a surveillance system that includes a plurality of video cameras (such as the video cameras14) each configured to provide video streams from a corresponding camera vantage point, the surveillance system including a surveillance system controller. The method74includes the surveillance system controller analyzing video streams provided by the plurality of video cameras in order to find a common object in a first video frame from a first video stream and a second video frame from a second video stream, as indicated at block76. The common object is tagged with an object identifier, as indicated at block78. View information for the tagged common object that includes view information from the camera vantage point of each of the first video stream and the second video stream is stored, as indicated at block80. A vantage point transformation is applied to the common object, the vantage point transformation transforming the view information for the common object from the camera vantage points of the first video stream and the second video stream to a common vantage point, as indicated at block82. The transformed view information of the common object from the common vantage point is rendered on a display, as indicated at block84.

In some instances, the method74may further include receiving user input to move the common vantage point to an updated common vantage point, and once moved, applying the vantage point transformation using the updated common vantage point and rendering the transformed view information of the common object from the updated common vantage point, as indicated at block86. The transformed view information of the common object from the common vantage point may be rendered at a first frame rate. The transformed view information of the common object from the updated common vantage point may be rendered at a second frame rate that is different from the first frame rate. The method74may further include rendering on the display indicators that indicate regions of a surveillance area and/or regions of the common object that are not within a camera FOV of any of the plurality of video cameras.

FIG.7is a schematic block diagram of an illustrative surveillance system controller90. The surveillance system controller90may be considered as being an example of the surveillance system controller12. The surveillance system controller90may be operably coupled with the plurality of video cameras14via a high speed wireless network such as but not limited to a 5G cellular network. When so provided, the surveillance system controller90includes a high speed input92that is configured to receive video streams from one or more of the plurality of video cameras via a high speed wireless network. In some cases, the surveillance system controller90also includes a high speed output94that is operably coupled with a high speed wireless network for outputting images such a videos, for example. In some instances, the high speed input92and the high speed output94may in combination be considered as forming a transceiver96, which can provide two-way communication. A memory98is operably coupled with the high speed input92and is configured to store received video streams. A controller100is operably coupled to the high speed input92and to the memory98. The controller100is configured to carry out a number of methods, such as those outlined inFIGS.8and9.

FIG.8is a flow diagram showing an illustrative method102that may be orchestrated by the controller100and thus carried out by the surveillance system controller90. The controller100may be configured to analyze each of the video streams in order to find a first common landmark in both a first video frame from a first video stream and a second video frame from a second video stream, as indicated at block104. When the first common landmark is present in both the first video frame from the first video stream and the second video frame from the second video stream, the controller100is configured to stitch together the first video frame from the first video stream and the second video frame from the second video stream and placing the stitched together image into a master image, as indicated at block106. When there is no first common landmark present in both the first video frame from the first video stream and the second video frame from the second video stream, the controller100is configured to populate the master image with the first video frame from the first video stream and the second video frame from the second video stream with each of the first video frames disposed within the master image at relative locations corresponding to physical locations of the views included in the first video frame and the second video frame, as indicated at block108. The master image is translated into a top view (or other birds eye view) of the master image, as indicated at block110.

In some instances, the controller100may also be configured to analyze each of the video streams in order to find a second common landmark in both the second video frame from the second video stream and a third video frame from a third video stream, as indicated at112. When the second common landmark is present in both the second video frame from the second video stream and the third video frame from the third video stream, the controller100may be configured to stitch together the second video frame from the second video stream and the third video frame from the third video stream and placing the stitched together image into the master image, as indicated at block114. When there is no second common landmark present in both the second video frame from the second video stream and the third video frame from the third video stream, the controller100may be further configured to populate the master image with the third video frame from the third video stream with the third video frame disposed within the master image at a relative location corresponding to a physical location of the view included in the third video frame, as indicated at block116.

In some cases, as indicated, at least some of the video cameras14may include video cameras that have an adjustable field of view (FOV). The controller100may be configured to send commands to adjust the FOV of one or more of the video cameras14having an adjustable FOV. When some of the video cameras14include mobile video cameras that may be secured relative to drones, the controller100may be configured to provide instructions to a particular mobile video camera (e.g. drone) to move to a particular location. In some cases, the controller100and/or the mobile video camera itself may be configured to estimate an optimal time for capturing a live video stream as the one or more of the mobile video cameras approach and/or pass beyond the particular location.

In some cases, the controller100may be configured to enable display of a location within the top view of the master image from any of a plurality of different vantage points. The plurality of different vantage points may include different locations on the ground as represented in an X-Y plane, for example. The plurality of different vantage points may include different heights in a Z plane that is orthogonal to the X-Y plane, as an example. The controller100may be configured to create and store a plurality of top views of master images based on video streams received from the plurality of video cameras over a period of time and to display a selected top view of a master image corresponding to a point in time within the period of time. The controller100may be configured to enable display of a location within the selected top view of the master image corresponding to the point in time from any of a plurality of different vantage points.

FIG.9is a flow diagram showing an illustrative method118of operating a surveillance system that includes a plurality of video cameras (such as the video cameras14) configured to provide video streams, the surveillance system including a surveillance system controller (such as the surveillance system controller12,90). The method118includes the surveillance system controller analyzing video streams provided by the plurality of video cameras in order to find a common landmark in a first video frame from a first video stream and a second video frame from a second video stream, as indicated at block120. When the common landmark is present in both the first video frame from the first video stream and the second video frame from the second video stream, the surveillance system controller stitches together the first video frame from the first video stream and the second video frame from the second video stream and placing the stitched together image into a master image, as indicated at block122. When there is no common landmark present in both the first video frame from the first video stream and the second video frame from the second video stream, the surveillance system controller populates the master image with the first video frame from the first video stream and the second video frame from the second video stream with each of the first video frames disposed within the master image at relative locations corresponding to physical locations of the views included in the first video frame and the second video frame, as indicated at block124. The surveillance system controller translates the master image into a top view of the master image, as indicated at block126.

FIG.10is a schematic block diagram of a drone128that is configured for use in a surveillance system (such as the surveillance system10) that includes a plurality of video cameras (such as the video cameras14) disposed within a surveillance area. The drone128includes a video camera130, a memory132, a transceiver134and a controller136that is operably coupled to the video camera130, the memory132and the transceiver134. In some cases, the video camera130may be considered as including a camera field of view (FOV)131. The transceiver134may be a cellular transceiver such as but not limited to a 5G cellular transceiver, for example. The controller136may be considered as being configured to control operation of the drone, including operation of the drone's flying capability as well as the video camera130.FIGS.11and12are flow diagrams showing illustrative methods that may be orchestrated by the controller136and thus carried out by the drone128.

FIG.11is a flow diagram showing an illustrative method138that the controller136of the drone128may be configured to carry out. A first frame of a video of an incident at a first location is captured using the video camera, as indicated at block140. A flight path to follow the incident is determined, as indicated at block142. The drone is flown along the flight path, as indicated at block144. A second frame of the video of the incident at a second location is captured using the video camera, wherein the second location is determined such that that there is an overlap in the camera Field of View (FOV) between the first frame of the video and the second frame of the video, as indicated at block146. The video is transmitted via the transceiver, as indicated at block148. In some cases, a frame rate of the video is dynamic and may be dependent on a speed of the drone flying along the flight path. In some cases, the frame rate may be dependent on an altitude of the drone flying along the flight path.

FIG.12is a flow diagram showing an illustrative method150that the controller136of the drone128may be configured to carry out. Instructions are received to fly to a particular location at which an incident is believed to be occurring, as indicated at block152. A first video of the incident is captured using the video camera, as indicated at block154. The controller136is configured to fly the drone to a second location away from the particular location to follow the incident, as indicated at block156. The controller136is configured to determine a time to capture a second video of the incident such that there is sufficient overlap between the first video and the second video to be able to stitch together the first video and the second video, as indicated at block158. The controller136is configured to capture a second video of the incident, as indicated at block160, and to stitch together the first video and the second video to create a stitched video, as indicated at block162. The controller136is configured to transmit the stitched video via the cellular transceiver (e.g. 5G transceiver), as indicated at block164.

FIG.13is a flow diagram showing an illustrative method166for determining when and how to stitch videos together. As seen, the method166can work with videos provided by any of fixed cameras168, PTZ cameras170and drone cameras172, particularly if the drone cameras172are carried by drones having an altitude that is similar to that of the fixed cameras168and/or the PTZ cameras170. In some cases, the drone cameras172having similar fields of view (FOV) may also play a part. At block174, cameras whose images can and should be stitched together are identified. Block174receives as inputs details regarding the surveillance area of interest, as indicated at block176, as well as physical locations of the cameras, as indicated at block178. The surveillance area of interest may be defined by the Field of View (FOV) of the particular birds eye view currently selected by an operator of the surveillance system10. The cameras whose images that should be stitched together may include all of the cameras that have a FOV that overlap with the field of view of the particular birds eye view currently selected by the operator of the surveillance system10. As the operator zooms in, the FOV of the birds eye view currently selected by the operator is reduced in size, and thus fewer cameras whose images should be stitched together may be identified. When fewer cameras are involved, a given set of processing resources can stitch together more frames, and the stitched together video may be rendered at a higher frame rate on an operator display.

In some cases, camera locations may be specified in terms of latitude and longitude, although other identifying criteria such as GPS coordinates may also be used. A camera view overlap module180determines which camera views are overlapping and which are not overlapping. Overlapping views are stitched together in the image stitching module182, resulting in a stitched image or stitched frame184. Non-overlapping views are aggregated into a final view, either as is, or transformed into a top or other birds eye view.

FIG.14is a flow diagram showing an illustrative method186of stitching views together, and may be considered as providing additional detail regarding the functionality of the camera view overlap module180and the image stitching module182. The illustrative method186includes as inputs the horizontal field of view (HFOV) and the vertical field of view (VFOV) of each of the video cameras, in accordance with specifications, as well as the physical locations of each of the cameras, as indicated at block188. Google maps or a similar engine may be used to identify landmarks and their physical locations, as indicated at block190. These are provided as inputs to block192, where a determination is made as to whether landmarks in the various camera views are overlapping and how much. This can be determined at least in part by identifying landmark locations within each camera view. If there is no overlap among landmarks in the various camera views, control passes to block194, where a master image is populated with the camera view details. Conversely, if landmarks do overlap in some of the camera views, control passes to decision block196where a determination is made as to whether there is substantial overlap. If so, control passes to block198where stitching occurs. At block200, a transformation to a top or other birds eye view is performed in accordance with known methods, and is viewed as indicated at block202.

FIG.15is a flow diagram showing an illustrative method204. As shown at206, a drone is flown from a position P1at time t1to a position P2at time t2. At time t1, the drone's field of view is from A to B. At time t2, the drone's field of view is from C to D. The method204includes, at position P1(as indicated at block208), ascertaining the drone's height and camera FOV, as indicated at block210. A first image is captured at position P1, as indicated at block212. A maximum range is calculated at block214and a position E (with reference to206) is determined as indicated at block216. In some cases, the position E may correspond to eighty percent of the maximum range divided by two. This enables some distance to be traveled between images while still allowing for sufficient overlap between successive images. At block218, a predicted position P2is determined. This may include as inputs the location (such as latitude and longitude) of the drone at time t1, as indicated at218a; the speed of movement as indicated at218b, the drone's altitude as indicated at218cand the yaw, pitch and roll of the drone, as indicated at218d. A second image is captured at the predicted position P2as indicated at block220. The first and second images are provided to image stitching algorithms222and then to a block224where a panoramic view is provided.

FIG.16is a schematic block diagram showing a series226of steps that may be carried out as part of conducting a search. At block228, a search query may be received from an operator. The search query may take a variety of forms, and could for example include a request to search for a male who is 60 to 75 inches tall, wearing a red shirt and carrying a black briefcase, for example. The search query could include a request to search for a yellow car that was seen passing through a particular intersection within a particular time window. The search query could include a request to search for gatherings of more than twenty people, for example. At block230, videos are searched to look for objects that match the search query. At block232, the location of matching objects are displayed on a transformed view from a common vantage point at a first frame rate. At block234, user input is received that zooms the common vantage point into a selected one of the matching objects. At block236, the selected matching object from the zoomed-in vantage point is displayed at a second frame rate. In some cases, the second frame rate may be higher than the first frame rate.

Having thus described several illustrative embodiments of the present disclosure, those of skill in the art will readily appreciate that yet other embodiments may be made and used within the scope of the claims hereto attached. It will be understood, however, that this disclosure is, in many respects, only illustrative. Changes may be made in details, particularly in matters of shape, size, arrangement of parts, and exclusion and order of steps, without exceeding the scope of the disclosure. The disclosure's scope is, of course, defined in the language in which the appended claims are expressed.