Patent Description:
A building surveillance system can include one or multiple cameras. Often, the number of cameras that need to be included in a building to capture all areas of a building or building premises is high. These numerous cameras result in an excessive cost of the building surveillance system, excessive installation time, etc. Furthermore, footage captured by the cameras may only be valuable in good visibility conditions. For example, at night or with various weather conditions (e.g., fog, rain, snow, etc.), the cameras may not capture footage of individuals that would normally be captured during good visibility conditions.

In addition to the costs associated with a high number of camera systems, surveillance systems may also be associated with high installation and calibration costs. Calibrating the camera systems can take a significant amount of time and technician expertise. For example, a significant amount of technician resources may be required to properly install and calibrate the surveillance system. In some instances, the cost of installing and calibrating a surveillance system may be greater than the cost of the surveillance system itself.

A building radar-camera surveillance system is disclosed in <NPL>. Documents relating to conventional camera calibration are <NPL>, <NPL>, <NPL>.

Various objects, aspects, features, and advantages of the disclosure will become more apparent and better understood by referring to the detailed description taken in conjunction with the accompanying drawings, in which like reference characters identify corresponding elements throughout.

Referring generally to the FIGURES, a building surveillance radar-camera system is shown, according to various exemplary embodiments. The radar-camera system can combine various artificial intelligence classification networks (e.g., Retinanet) with building cameras (e.g., pan, tilt, and zoom (PTZ) cameras) and a ground radar system to facilitate surveillance for a building premise. The system can be autonomous and require little or no human control or involvement in calibration. When the radar system detects a moving object (e.g., a person, an animal a car, etc.), the system can be configured to control cameras to capture images of the moving object and classify the moving object, e.g., determine whether the object is a person, a vehicle, an animal, etc. Using both a radar system and a camera system can solve problems in conventional surveillance systems that require a high number of static cameras and/or human security personal.

A conventional surveillance system can suffer from a high false alarm rate which may be especially prevalent in systems with moving PTZ cameras. Furthermore, the conventional system may lack high quality object detection and classification; these conventional systems may generate the same alert for a person as it would for an animal. Furthermore, the conventional system may perform poorly in various poor vision environmental conditions, e.g., at night, in heavy fog, etc..

The surveillance system described herein can be configured to utilize improved classification networks, can include improved object tracking for controlling a PTZ camera from a central system (e.g., a server), can perform internal camera parameter calibration, external camera parameter calibration. The internal parameters may be parameters that depend upon location and/or orientation of a camera. The internal parameters may be parameters of the camera that a used to move or zoom that camera e.g., focal length. The external parameters may be parameters between the camera and the outside world, for example, a translation between points identified in a camera space to a world space. To perform the external calibration, a system may use or determine corresponding pairs of data between the camera space and the world space, e.g., a correspondence of points of the camera space (detected via the camera) and the world space (detected via the radar system).

The system discussed herein is configured to accurately detect, classify, and/or track objects in real-time, and estimate their real-world position. By using the radar system and laser cameras, the system discussed herein overcomes various issues faced by conventional surveillance systems. The system described herein may be a partially or fully automated surveillance system that uses a radar system and a small number of PTZ cameras that can replace a high number of static cameras and/or human security personal in a conventional surveillance system.

Referring now to <FIG>, a building <NUM> with a security camera <NUM> and a parking lot <NUM> is shown, according to an exemplary embodiment. The building <NUM> is a multi-story commercial building surrounded by, or near, the parking lot <NUM> but can be any type of building in some embodiments. The building <NUM> may be a school, a hospital, a store, a place of business, a residence, a hotel, an office building, an apartment complex, etc. The building <NUM> can be associated with the parking lot <NUM>.

Both the building <NUM> and the parking lot <NUM> are at least partially in the field of view of the security camera <NUM>. In some embodiments, multiple security cameras <NUM> may be used to capture the entire building <NUM> and parking lot <NUM> not in (or in to create multiple angles of overlapping or the same field of view) the field of view of a single security camera <NUM>. The parking lot <NUM> can be used by one or more vehicles <NUM> where the vehicles <NUM> can be either stationary or moving (e.g. busses, cars, trucks, delivery vehicles). The building <NUM> and parking lot <NUM> can be further used by one or more pedestrians <NUM> who can traverse the parking lot <NUM> and/or enter and/or exit the building <NUM>. The building <NUM> may be further surrounded, or partially surrounded, by a sidewalk <NUM> to facilitate the foot traffic of one or more pedestrians <NUM>, facilitate deliveries, etc. In other embodiments, the building <NUM> may be one of many buildings belonging to a single industrial park, shopping mall, or commercial park having a common parking lot and security camera <NUM>. In another embodiment, the building <NUM> may be a residential building or multiple residential buildings that share a common roadway or parking lot.

The building <NUM> is shown to include a door <NUM> and multiple windows <NUM>. An access control system can be implemented within the building <NUM> to secure these potential entrance ways of the building <NUM>. For example, badge readers can be positioned outside the door <NUM> to restrict access to the building <NUM>. The pedestrians <NUM> can each be associated with access badges that they can utilize with the access control system to gain access to the building <NUM> through the door <NUM>. Furthermore, other interior doors within the building <NUM> can include access readers. In some embodiments, the doors are secured through biometric information, e.g., facial recognition, fingerprint scanners, etc. The access control system can generate events, e.g., an indication that a particular user or particular badge has interacted with the door. Furthermore, if the door <NUM> is forced open, the access control system, via door sensor, can detect the door forced open (DFO) event.

The windows <NUM> can be secured by the access control system via burglar alarm sensors. These sensors can be configured to measure vibrations associated with the window <NUM>. If vibration patterns or levels of vibrations are sensed by the sensors of the window <NUM>, a burglar alarm can be generated by the access control system for the window <NUM>.

Referring now to <FIG>, a block diagram of an ACS <NUM> is shown, according to an exemplary embodiment. The ACS <NUM> can be implemented in any of the building <NUM> as described with reference to <FIG>. The ACS <NUM> is shown to include doors <NUM>. Each of the doors <NUM> is associated with a door lock <NUM>, an access reader module <NUM>, and one or more door sensors <NUM>. The door locks <NUM>, the access reader modules <NUM>, and the door sensors <NUM> may be connected to access controllers <NUM>. The access controllers <NUM> may be connected to a network switch <NUM> that directs signals, according to the configuration of the ACS <NUM>, through network connections <NUM> (e.g., physical wires or wireless communications links) interconnecting the access controllers <NUM> to an ACS server <NUM>. The ACS server <NUM> may be connected to an end-user terminal or interface <NUM> through network switch <NUM> and the network connections <NUM>.

The ACS <NUM> can be configured to grant or deny access to a controlled or secured area. For example, a person <NUM> may approach the access reader module <NUM> and present credentials, such as an access card. The access reader module <NUM> may read the access card to identify a card ID or user ID associated with the access card. The card ID or user ID may be sent from the access reader module <NUM> to the access controller <NUM>, which determines whether to unlock the door lock <NUM> or open the door <NUM> based on whether the person <NUM> associated with the card ID or user ID has permission to access the controlled or secured area.

Referring now to <FIG>, a radar-camera system <NUM> is shown, according to an exemplary embodiment. The radar-camera system <NUM> can be implemented in the building <NUM> of <FIG> and configured to perform surveillance at the building <NUM>, in some embodiments. System <NUM> can be implemented in a commercial building, in an oil and/or gas production site, in a utility substation, at a car lot, in a neighborhood, in an airport, in mass a transit system, in an entire city, in a smart building, etc. The system <NUM> can be a combination of an autonomous PTZ camera system and a radar system. The system <NUM> is configured to detect, classify, track, and/or estimate real-world positions for objects in real-time, in some embodiments. The system <NUM> can monitor and/or detect objects, understand and gather information on the objects (e.g., classify the objects), and/or respond to the classification (e.g., raise an alarm, unlock a door, etc.). The images of the cameras may be high resolution, containing a high number of pixels. A higher number of pixels may result in better detection, tracking, and classification by the system <NUM>. This may also enable or improve facial recognition performed by the system <NUM>.

The system <NUM> can be a partial or fully autonomous surveillance system. Surveillance and video analytics is an advantageous component in a building security system in some embodiments. Since in many systems, the number of security cameras grows exponentially over time to cover as many views of a building as possible, having a human constantly watch and understand footage of the building that is captured by the security cameras can be difficult. It may not be feasible to have a human monitor every camera of a building since the number of cameras may be high. The system <NUM> can address these issues by automatically controlling building security cameras and/or analyzing security footage, according to some embodiments.

Some image analysis systems suffer from high false alarm rates. High false alarm rates can result from moving PTZ cameras, a lack of high quality object classification (e.g., an animal and a person may generate the same alert), and/or poor vision conditions (e.g., night, fog, etc.). Some video analytics may be based on change detection, in some embodiments. A change detection system may be a system that detects objects based elements in an image changing with respect to a background. However, bad weather, clouds, camera noise and especially moving cameras can limit the quality and robustness of change detection. Due to the limitations of change detection video analysis, a change detection system may require substantial human supervision. Artificial Intelligence (AI) based classification can run on individual frames, may not be sensitive to camera movements, and can be robust to outdoor conditions (e.g., shadows, rain, etc.), all areas in which change detection systems may fall short. Furthermore, based on the calibration between the radar system and the camera system, objects and their sizes detected in the images to help reduce false alarms. For example, if a user is detected in in an unauthorized area but, based on the calibration, the user is a taller than a predefined amount as can be determined via the calibration, the system can determine that the user is not properly classified and that the object is a different class (e.g., a tree) and thus an alarm can be stopped from being triggered.

The system <NUM> is configured to implement one or more of AI algorithms, a radar system, laser cameras, and/or powerful graphics processing units, in some embodiments. These components can allow the system <NUM> to implement a partial and/or fully autonomous surveillance system, in some embodiments. The system <NUM> can be configured to implement AI algorithms to perform object detection and classification with deep neural networks (DNNs). The system <NUM> can include a GPU configured to implement DNNs. The performance of object detection and classification by the system <NUM> can be high even for moving cameras.

The system <NUM> is further configured to include a radar system, in some embodiments. The radar system may provide a cost efficient and accurate system that is not limited by poor weather conditions (e.g., fog, night, etc.). Furthermore, the system <NUM> can include laser cameras. In some embodiments, the laser cameras are infrared (IR) laser cameras configured to view objects at night up to <NUM> meters. In some embodiments, the laser cameras and the radar system are used with millimeter wave cameras or other vision system. GPU computational power enables the system <NUM> to run DNNs at affordable prices, GPUs may provide much higher image processing power than CPUs.

The system <NUM> is configured to control the orientation (e.g., the pan, tilt, and/or zoom) based on radar detections in some embodiments. For example, when the system <NUM> detects an object via the radar system, the system <NUM> can control an appropriate camera to be pointed at the object and can be configured to utilize artificial intelligence to track and classify the object (at a reliability better than humans). Furthermore, the system <NUM> is configured to facilitate a handover of an object from a first camera to a second camera if the object is moving from a view space of the first camera into the view space of a second camera. Since the cameras <NUM> and/or <NUM> can be controlled to track an object, object tracking over a wide range can be achieved.

The system <NUM> is shown to include a security system manager <NUM>. The manager <NUM> can be a central system of the system <NUM> configured to communicate and/or control a radar system <NUM> and/or security cameras <NUM> and/or <NUM>, according to some embodiments. The manager <NUM> can be implemented on premises within the building <NUM> of <FIG> and/or off-premises in a cloud system, e.g., a MICROSOFT AZURE system, an AMAZON WEB SERVICES (AWS), and/or any other remote web server and/or system.

The manager <NUM> is shown to include a processing circuit <NUM>. The processing circuit <NUM> can be any central purpose processor (CPU), graphics processing unit (GPU), application specific integrated circuit (ASIC), and/or any other component for performing computations combined with memory devices. The processing circuit <NUM> is shown to include a processor <NUM> and a memory <NUM>. In some embodiments, the security system manager <NUM> is made up of multiple processing circuits that are distributed across multiple computing systems, servers, controllers, etc. However, as an illustrative embodiment, the security system manager <NUM> is described with a single processing circuit, the processing circuit <NUM> which can be one or multiple processing circuits.

The processing circuit <NUM> is shown to include a processor <NUM> and a memory <NUM>. The processing circuit <NUM> can include any number of processing devices and/or memory devices. The processor <NUM> can be implemented as a general purpose processor, an application specific integrated circuit (ASIC), one or more field programmable gate arrays (FPGAs), a group of processing components, or other suitable electronic processing components. The memory <NUM> (e.g., memory, memory unit, storage device, etc.) may include one or more devices (e.g., RAM, ROM, Flash memory, hard disk storage, etc.) for storing data and/or computer code for completing or facilitating the various processes, layers and modules described in the present application. The memory <NUM> can be or include volatile memory and/or non-volatile memory.

The memory <NUM> can include object code components, script components, or any other type of information structure for supporting the various activities and information structures described in the present application. According to some embodiments, the memory <NUM> is communicably connected to the processor <NUM> via the processing circuit <NUM> and can include computer code for executing (e.g., by the processing circuit <NUM> and/or the processor <NUM>) one or more processes of functionality described herein.

The radar system <NUM> may be a radar system deployed at the building <NUM> of <FIG> configured to communicate with the manager <NUM>, according to some embodiments. The radar system <NUM> may a MAGOS radar system. The radar system <NUM> can utilize Multiple Input Multiple Output (MIMO) and digital beam forming technology to cover an area of over <NUM>,<NUM> m<NUM> (> <NUM> acres) with a detection range of <NUM>m for a person walking and <NUM>m for vehicle and/or boat. The radar system <NUM> is, in some embodiments, a low power system that consumes an extremely lower amount of power and/or may be a cost effective radar solution. A MAGOS radar system may consume < <NUM>W and have small form factor. The MAGOS radar system can include an ultra-high range resolution of <NUM> meters giving it excellent performance in cluttered environments. The MAGOS radar system may be small in size, low in power consumption, and low in weight make it simple to install and rendering it ideal as a deployable system as well. The radar system can be configured to provide wide tracking track objects during the day and/or night and/or in all types of weather. Some applications may not include the radar system <NUM> but still may require camera tracking with the cameras <NUM> and/or <NUM>.

The radar system <NUM> can identify the locations of objects and track the objects as they move. The radar system <NUM> may identify the locations of the objects as coordinate values and/or angles and distances from the radar system <NUM> on a world plane. In some embodiments, the systems and methods discussed herein can utilize other world plane based systems, e.g., an electric fence, an access control system (e.g., as described with reference to <FIG>), pressure sensors located around a building, etc..

The cameras <NUM> and <NUM> may be security cameras that are movable, i.e., the cameras <NUM> and/or <NUM> are configured to pan, tilt, or zoom (e.g., x<NUM> zoom), according to some embodiments. The cameras <NUM> and/or <NUM> are configured to capture high resolution images and/or video in some embodiments. The manager <NUM> can control the orientation of the cameras <NUM> and/or <NUM>. In some embodiments, the cameras <NUM> and/or <NUM> are infrared (IR) cameras that can be capture high quality images at night at long ranges. In some embodiments, the security cameras <NUM> and/or <NUM> are positioned in various location entrances, on rooftops, on outer walls, on grounds of a facility, in various locations to capture images and/or video of a user, animal, and/or vehicle walking, moving, and/or driving. Although the system <NUM> is shown to include two cameras, the system <NUM> can include any number of cameras.

The memory <NUM> of the manager <NUM> is shown to include a radar system manager <NUM>, a camera manager <NUM>, and a calibrator <NUM>. The camera manager <NUM> can be configured to detect objects within a frame and/or frames captured by the cameras <NUM> and/or <NUM>. The camera manager <NUM> is configured to classify each of the detected objects and/or track the objects if the objects are moving, in some embodiments. The camera manager <NUM> is configured to implement classification networks (e.g., DNNs) to perform the object detection and/or classification in some embodiments. The camera manager <NUM> is configured to implement a deep learning framework to track objects in video captured by cameras <NUM> and/or <NUM> in some embodiments. The camera manager <NUM> can perform deep object-detection on each frame and use temporal information to create consistent tracks and remove false detections.

The camera manager <NUM> can be configured to track and identify objects via a fusion of radar data of the radar system <NUM> and/or images captured by cameras <NUM> and/or <NUM>. The camera manager <NUM> can receive radar data, e.g., from radar system manager <NUM>, and control cameras <NUM> and/or <NUM> based on the radar data. For example, if radar system <NUM> detects a moving person, car, and/or animal at specific coordinates, camera manager <NUM> can control the movement of cameras <NUM> and/or <NUM> to move (pan, tilt, and/or zoom) to view the moving object. The camera manager <NUM> can detect the object and track the object, continuing to move the cameras <NUM> and/or <NUM> to keep the object within a frame captured by the cameras <NUM> and/or <NUM>.

The camera manager <NUM> can control the cameras <NUM> and/or <NUM> so that the object is kept in the middle of the frame. In some embodiments, the camera manager <NUM> is configured to classify the moving object, e.g., as a person, as an animal, as a car, as a boat, etc. The camera manager <NUM> can be configured to perform error correction and/or filtering to improve image classification and/or tracking. The camera manger <NUM> can perform error correction and filtering for object type classification confidence, object width in meters, speed of objects in meters per second, and/or location of an object (e.g., latitude and/or longitude). The error correction and/or filtering can work with and/or after all of the steps of the processes described herein. There may be a balance between image detection, classification, and/or tracking speed (e.g., whether the performance is real-time or near real-time) and accuracy. In some embodiments, the camera manger <NUM> is configured to handle four video streams in parallel but can be configured to handle any number of video streams.

The calibrator <NUM> can be configured to perform semi-automatic and/or automatic calibration of cameras <NUM> and/or <NUM>. The calibrator <NUM> can be configured to perform a camera-to-world calibration method for the cameras <NUM> and <NUM>. The calibration may be agnostic to zoom levels for the cameras <NUM> and <NUM>. The calibrator <NUM> can be configured to solve an optimization problem which maps between the visual objects captured by the cameras <NUM> and/or <NUM> and a world space of the radar system <NUM>. The optimization can be performed by only using correspondences between the camera orientation and world coordinates. This can remove the need to calibrate internal camera parameters although in some embodiments, the internal camera parameters may still be calibrated.

Furthermore, the calibrator <NUM> can be configured to perform a calibration for the various focal lengths of the cameras <NUM> and/or <NUM> automatically without a man-in-the-loop. This enables a moving object to be kept in the center of the image from a remote machine, overcoming communication delays. The calibration performed by the calibrator <NUM> can be highly accurate and can help fully and/or partially automate the system <NUM>; the calibration can improve the object classification and tracking of the camera manager <NUM>.

The radar system manager <NUM> can be configured to communicate with and/or control the radar system <NUM>. The radar system manager <NUM> can be configured to receive coordinates of moving objects from radar system <NUM>. In some embodiments, the radar system manager <NUM> is configured to generate and/or store a world view, coordinate based mapping of various objects detected by the radar system <NUM>. The radar system manager <NUM> is configured to provide the world view coordinates to camera manager <NUM> so that camera manager <NUM> can control cameras <NUM> and/or <NUM> and to calibrator <NUM> for calibration performed by calibrator <NUM>, according to an exemplary embodiment. In some embodiments, the radar system manager <NUM> is configured to store and record a track of an object by recording the position over time.

The manager <NUM>, the radar system <NUM>, and/or the cameras <NUM> and/or <NUM> can share bounding boxes. A bounding box may be an indication of a group of pixels in an image that are pixels of a particular object, e.g., a person, a car, a boat, etc. The bounding box can be based on Java Script Object Notation (JSON). Furthermore, a link to a live video stream of cameras <NUM> and/or <NUM> can be embedded in boxes inside a web-container.

The system <NUM> can further include and/or integrate with a video management system (VMS) and/or physical security information management (PSIM) system. For example, the system <NUM> can retrieve, and/or make available, a video stream of cameras <NUM> and/or <NUM> with an embedded box around a detected object in a web-container (e.g., the images shown in <FIG>). Furthermore, the system <NUM> can receive and/or send bounding box information (e.g., an indication of the object, human, animal, vehicle, with a time tag to overlay part (e.g., the top) of the video). In some embodiments, the bounding box can be embedded within a real-time streaming protocol (RTSP). The system <NUM> can act as RTSP proxy and send synchronized bounding box metadata to various integrated systems.

Still referring to <FIG>, the system <NUM> is shown to include a central security system manager <NUM>. In some embodiments, the system manager <NUM> may be distributed to edge devices. For example, camera <NUM>, camera <NUM>, and/or radar system <NUM> can run some and/or all of the functions of manager <NUM>. This can remove long communication delays if any are present and improve the real-time response of the system <NUM>. Furthermore, the number and/or power of manager <NUM>, can be increased, e.g., adding more and/or more powerful GPUs. This can increase object classification reliability and/or can increase the number of types of objects that the manager <NUM> can classify.

Referring now to <FIG>, images <NUM>-<NUM> illustrating the system <NUM> tracking a person is shown, according to an exemplary embodiment. In image <NUM> the cameras <NUM> and <NUM> are shown. In image <NUM>, the camera <NUM> is controlled by the camera manager <NUM> to capture images of the user. The individual can be detected by the radar system <NUM>. Based on the detection of the radar system <NUM>, the camera manager <NUM> is configured to control (e.g., control pan, tilt, and/or zoom) the camera <NUM> to capture images of the individual. As the individual moves, the camera manager <NUM> can cause the camera <NUM> to pick up the individual, performing a handover between the cameras <NUM> and <NUM>. In some embodiments, the system <NUM> does not respond to certain objects e.g., animals if the system <NUM> is configured to only track animals. In some embodiments, the system <NUM> is configured to only detect and/or track objects that are within a predefine range of a building.

Each of the cameras <NUM> and <NUM> can scan their environment and report on specific targets. The system <NUM> can be configured to utilize a centralized computer which analysis every frame captured by the cameras <NUM> and <NUM> in real-time and understand what the object is. The radar system can efficiently detect both static and moving objects. Based on radar and/or camera detection, the manager <NUM> can place the moving object on a map and raise an alarm. Furthermore, based on the detection of the radar and/or solely based on camera detection, the manager <NUM> can generate a world position estimate of a moving object.

Referring now to <FIG>, a process <NUM> is shown for detecting and tracking an object. The process <NUM> can be performed by the camera manager <NUM> with a classification model (e.g., a neural network, a faster regional convolutional neural network (R-CNN), etc.), a Kalman filter, and/or matching algorithms (e.g., a Hungarian matching algorithm), and/or camera movement compensation algorithms. The camera manager <NUM> can be configured to perform the process <NUM>. The process <NUM> may be similar to the processes described with reference to <FIG>.

In step <NUM>, the camera manager <NUM> can generate proposals for targets in an image captured by a camera. The proposals may be groups of pixels, e.g., pixels bound by a box, that the camera manger <NUM> determines should be classified. The camera manager <NUM> can utilize a classification model to identify the regions of the image that should be classified as one of a set of known objects (e.g., vehicles, people, animals, plants, etc.). In some embodiments, the camera manager <NUM> is configured to use a faster R-CNN. Using a faster R-CNN may result in a small number of pixel area proposals. The camera manager <NUM> can use various object classification algorithms, e.g., decision trees, Bayesian networks, etc. to classify the objects of the image proposals. Region proposals <NUM> and <NUM> illustrate areas of an image that the camera manager <NUM> may identify as pertaining to a particular target to be tracked by the camera manager <NUM>.

In step <NUM>, the camera manager <NUM> can predict next locations of the objects within the image with a Kalman. The next locations may be a prediction, the prediction locations <NUM> and <NUM>, of where the objects represented by the region proposals <NUM> and <NUM> will move to in the future, e.g., in a next frame. The Kalman filter can use one or multiple previous frames, object detections, and previous predictions to generate the predicted next locations <NUM> and <NUM>.

In step <NUM>, the camera manager <NUM> can score the prediction of the Kalman filter by generating a score between each of the predictions <NUM> and <NUM> and actual locations of the objects in a current frame (a frame subsequent to the frame used to generate the prediction locations <NUM> and <NUM>). The score may indicate the likelihood that a detection of an object in a current image is a previously detected object or a new object. For example, for a particular object detected in a current frame, the score between the predicted next location <NUM> and the current detection of the object may indicate the likelihood that the current detection of the object is the same object as the object associated with the prediction location <NUM>. This allows the camera manager <NUM> to track objects through multiple frames and identify new objects. In some embodiments, a matrix of scores is generated to associate a score between each prediction based on a first frame of the step <NUM> and each actual location of the objects of the subsequent frame.

In some embodiments, the scores are based on a comparison of locations of the next locations and the actual locations. For example, the scores can be distances between the predicted next locations and the actual locations. If the next locations and actual locations are represented as pixel areas, the scores can be Intersection-Over-Union (IoU) scores, i.e., an area of intersection of the pixel areas divided by an area of union of the pixel areas. Furthermore, the scores can be based on object classifications. For example, a predicted next location for a human may be scored with an actual location of a classified human differently than the predicted next location for the human an a second actual location of a car. In some embodiments, one or all of the scoring techniques can be used to generate the scores.

In step <NUM>, the camera manager <NUM> can match the next locations and the actual locations via a matching algorithm based on the scores. If a match is determined between an object of an actual location and a predicted next location, the object of the actual location and the predicted next location can be determined to be the same object and thus the camera manager <NUM> can maintain a track of the object. Such a determination can be performed by the camera manager <NUM> for each of multiple objects in the image. If an object with an actual location does not match any predicted next location of the step <NUM>, the camera manager <NUM> can determine that the object is a new object and can begin tracking the new object. The matching algorithm can be any type of algorithm. For example, the matching algorithm may be a Hungarian matching algorithm.

Based on the matches determined in the step <NUM>, in step <NUM>, camera manager <NUM> can update the Kalman filter used to predict the next locations in step <NUM>. For example, the tracks used as input into the Kalman filter to generate the next locations <NUM> and <NUM> can be based on the tracks determined via the step <NUM>, i.e., a sequence of actual locations of an identified object through multiple frames. The process of predicting the next location of an object (the step <NUM>), scoring the next location of the object with actual detected locations of the objects (the step <NUM>), determining whether the next location and the actual location area associated with the same object based on the score (the step <NUM>), and updating the Kalman filter (the step <NUM>) can be performed iteratively such that objects are tracked through time and new objects are identified as the new objects enter the frames.

Referring now to <FIG>, the camera <NUM> can be controlled by the camera manager <NUM> to maintain a target within the image based on the prediction of the Kalman filter. As can be seen, the camera is adjusted so that tracked objects are kept within the frame. As shown in <FIG>, two objects are shown represented by the predicted next locations <NUM> and <NUM> and the region proposals <NUM> and <NUM>. However, any number of objects can be detected and tracked. The camera manager <NUM> can be configured to pan, tilt, or zoom the camera <NUM> such that the predicted next locations <NUM> and <NUM> are centered within the frame of the camera. When multiple simultaneous objects are detected, multiple cameras (e.g., two cameras) can point towards both objects or one camera can point to one object and another camera can point to another object.

Because the camera <NUM> is moved to a new position, the Kalman filter used to predict the next locations <NUM> and <NUM> may become inaccurate or lose the objects the Kalman filter is tracking. To compensate for the camera movement, the camera manager <NUM> can compensate for the movement of a camera in the Kalman filter with a homography. In some embodiments, a step <NUM> of the process <NUM> described with reference to <FIG> includes operating, by the camera manager <NUM>, the camera <NUM> to center the next locations <NUM> and <NUM> and perform motion compensation within the Kalman filter with a homography. The motion compensation may include adjusting one or more parameters of the Kalman filter based on the homography. In some embodiments, the camera manager <NUM> translates points of the frame from the new camera position back to a previous camera position or a "first frame" with the homography. In this regard, the Kalman filter may operate according to a static reference.

Referring now to <FIG>, a process <NUM> for object classification and tracking is shown that can be performed by the security system manager <NUM>, according to an exemplary embodiment. While any computing device described herein can be configured to perform the process <NUM>, e.g., the security system manager <NUM>, the process <NUM> is described with reference to the camera manager <NUM>. The process <NUM> is shown to include four steps, all of which can be performed by the camera manager <NUM>. In step <NUM>, the camera manager <NUM> can receive an image including one or more objects. In some embodiments, the camera manager <NUM> receives the image from a database or from one of the cameras <NUM> and/or <NUM>.

In step <NUM>, the camera manager <NUM> can detect the one or more objects and classify the one or more objects with a classification model. For example, based on the images received in the step <NUM>, the camera manager <NUM> can detect an object, either stationary or moving, and classify the object, e.g., classify the object as a person, a vehicle, an animal, etc. The camera manager <NUM> can implement a neural network, e.g., a faster R-CNN to perform the object detection and/or classification. The step <NUM> is described in greater detail in <FIG>.

In step <NUM>, the camera manager <NUM> can track the detected and classified object of step <NUM>. The camera manager <NUM> can generate a prediction of where in an image captured by a camera the object will be and/or can control the camera to keep the object within the image (e.g., keep the object at the center of the image captured by the camera). The camera manger <NUM> can implement a Kalman filter to perform the object tracking. Specifically, the camera manager <NUM> can generate a prediction of where the object will be based on the Kalman filter. The step <NUM> is described in greater detail in <FIG>.

In step <NUM>, the camera manager <NUM> can perform filtering of the tracks of the objects generated in the step <NUM> (or over multiple iterations of the steps <NUM> and <NUM>) based on speed and size of the objects. The camera manager <NUM> can perform the step <NUM> to identify a speed of the object by averaging a speed of the object over time and then normalizing the speed of the object based on a size of the object to account for a distance of the object from the camera. The normalization can take into account the fact that a smaller object is farther away from the camera and is traveling at a greater speed than an object closer to the camera even if the speed reported by the Kalman filter for both objects is the same. The tracks filtered by the camera manager <NUM> may be static tracks, i.e., tracks of an object that do not change in speed by a predefined amount and/or tracks of an object where the object does not change in size by a predefined amount. The tracks may be tracks created by the camera manager <NUM> based on the tracked object and may be a sequence of positions of the object over time. The step <NUM> is described in further detail as its own process in <FIG>.

Referring now to <FIG>, the step <NUM> is shown in greater detail as a process that the camera manager <NUM> is configured to perform, according to an exemplary embodiment. The process can include multiple steps, i.e., the steps <NUM>-<NUM>. While any computing device described herein can be configured to perform the step <NUM>, e.g., the security system manager <NUM>, the step <NUM> is described with reference to the camera manager <NUM>. In step <NUM>, the camera manager <NUM> can receive multiple images, each image including one or more objects. The images can be footage of a camera captured over time, i.e., each image may be a video frame. In some embodiments, the camera manager <NUM> receives the image from a database or from one of the cameras <NUM> and/or <NUM>.

The step <NUM> can be performed with a tensor flow model and/or can be performed on GPUs e.g., 1080TI GPUs. The system <NUM> can optimize the performance of the 1080TI with optimization based on visual molecular dynamics (VMD) while camera is not moving. While there is a tradeoff between speed and quality, the step <NUM> can be performed on video frames at <NUM> FPS with high definition video (HD) by tuning parameters for one GPU. For four cameras operating simultaneously each at <NUM> FPS (HD video), two GPUs can be used by the camera manager <NUM>.

In the step <NUM>, the camera manager <NUM> can analyze each image of the images received in the step <NUM> according to sub-steps <NUM>-<NUM>. The steps <NUM>-<NUM> can be performed iteratively, for example, if the images of the step <NUM> are received one by one, the sub-steps <NUM>-<NUM> can be performed each time a new image is received.

In the sub-step <NUM>, the camera manager <NUM> can identify multiple region proposals within the images, each of the region proposals corresponding to one of the one or more objects. The proposal regions may be areas (e.g., groups of pixels) within an image where the camera manager <NUM> determines an object of interest may be present. The camera manager <NUM> can use a neural network e.g., a Tensorflow model, to perform the object detection.

Based on the regions detected in the step <NUM>, the camera manager <NUM> can classify each region proposal into one of several target classes (e.g., human, animal, car, etc.). In some embodiments, the camera manager applies a classification filter to the classes. For example, the camera manager <NUM> can include a filter that indicates a hierarchy of classes and filters according to the hierarchy of classes. For example, a top level class may be a vehicle class while the vehicle class is associated with a set of sub-classes, e.g., a truck class, a sedan class, a cross-over vehicle class, etc. For any region proposal classified as one of the sub-classes, based on the filter, the camera manager <NUM> can apply the top level class. For example, if a truck is identified by the camera manager <NUM>, the camera manager <NUM> can assign the truck the vehicle class.

Finally, in the sub-step <NUM>, the camera manager <NUM> can fine tune the region proposals of the classified regions of the sub-step <NUM> to and generate a bounding box for each region proposal. The bounding box may be a box that surrounds the proposal region and provides an indication of the classification of the proposal region. In some embodiments, the region proposals can be reduced from a first area as identified in step <NUM> to a second area. The camera manager <NUM> can generate a box and apply the box around the second area. The image analyzed in the sub-steps <NUM>-<NUM> can be presented to a user with the box overlaid such that information regarding the objects is presented to an operator. Examples of objects with overlay bounding boxes is provided in <FIG>.

Referring now to <FIG>, the step <NUM> is shown in greater detail as a process that the camera manager <NUM> can perform, according to an exemplary embodiment. The process can include multiple steps, i.e., the steps <NUM>-<NUM>. The step <NUM> can track a detected object of the step <NUM>. The step <NUM> can be the same as, or similar to, the process <NUM> as described with reference to <FIG>. While any computing device described herein can be configured to perform the step <NUM>, e.g., the security system manager <NUM>, the step <NUM> is described with reference to the camera manager <NUM>.

In step <NUM>, the camera manager <NUM> can perform camera motion compensation using a homography. This can allow the Kalman filter to understand the position of the detected object even if the camera is moving. A homography is described in further detail in <FIG> and elsewhere herein. In some embodiments, the camera manager <NUM> can transform all points in an image when the camera is moving to a reference frame such that the Kalman filter operates according to a stable reference point. In some embodiments, the camera manager <NUM> adjusts parameters, e.g., internal states of the Kalman filter, to account for movement of the camera.

In step <NUM>, the camera manager <NUM> can predict an object bounding box using a Kalman filter. The prediction of the object bounding box may be future location for the bounding box based on a current location of the bounding box. This may be a prediction of the movement of the object represented by the bounding box. The prediction by the Kalman filter can be made based on one or multiple past known locations of the object (e.g., past bounding boxes). The Kalman filter can track one or multiple different objects, generating a predicted location for each. The prediction of the Kalman filter may not be affected by movement of the camera since internal states of the Kalman filter can be compensated for using movement of the camera with the homography as described in step <NUM>.

In step <NUM>, the camera manager <NUM> can determine a similarity between a predicted tracks of the object, e.g., the predicted locations of the bounding boxes, and actual tracks of the one or more objects, e.g., new bounding box locations representing an actual location of the objects within a subsequent image. The similarity can be determined with intersection by determining IoU values. For two bounding boxes, a predicted bounding box and an actual subsequently determined bounding box, the union may be the total area of the overlapping and non-overlapping portions of the bounding boxes summed together. The intersection may be the area of only the overlapping portions of the two bounding boxes. The IoU may be the intersection divided by the union. The higher the value of the IoU, the better the prediction of the Kalman filter and the higher the probability that the object of the predicted location is the same object as in the subsequent image. For example, an IoU over <NUM> may be considered to be a correct IoU or an IoU that confirms that the object of the prediction and the object of the subsequent image are the same object.

In step <NUM>, the camera manager <NUM> can match the objects and tracks using the similarity scores of the step <NUM> with a matching algorithm. The matching can identify the objects across multiple frames and further identify any new objects (e.g., an object of a new frame that is not matched with any previous objects). The matching algorithm can be a marriage/Hungarian algorithm. In some embodiments, rather than, or in addition to, simply identify whether the IoU of the step <NUM> is above a predefined amount, a matching algorithm can be used. This can allow for tracking of the objects through multiple frames even when the objects are partially occluded or disappear from the frames for a period of time.

Referring now to <FIG>, the step <NUM> is shown in greater detail as a process that the camera manager <NUM> can perform, according to an exemplary embodiment. The process can include multiple steps, i.e., the steps <NUM>-<NUM>. While any computing device described herein can be configured to perform the step <NUM>, e.g., the security system manager <NUM>, the step <NUM> is described with reference to the camera manager <NUM>. In some embodiments, the step <NUM> is only performed for static objects, i.e., objects which are not changing in size. This is indicative that the object is not traveling towards the camera and is instead traveling perpendicular to the camera.

In step <NUM>, the camera manager <NUM> can acquire a speed of the detected object from an internal state of the Kalman filter. As described in <FIG> and <FIG>, the object can be tracked by the Kalman filter. The camera manager <NUM> can retrieve a predicted speed of the object from the Kalman filter for each frame of a sequence of frames analyzed by the camera manager <NUM>. The values of the speed may not be affected by movement of the camera capturing the image. As previously described, a homography can be used to compensate for camera movement by adjusting internal parameters of the Kalman filter.

In step <NUM>, the speed can be collected over time and averaged in step <NUM>. The speed retrieved from the Kalman filter for each image of the sequence of images can be averaged to generate the average speed for the object. In step <NUM>, the camera manager <NUM> can normalize the average speed determined in the step <NUM> based on the size of the detected object in step <NUM>. The speed may be relative to the size of the object since distant objects may move a fewer number of pixels a second that a closer object. In this regard, the average speed can be normalized to a pixel area of the object. The pixel area may be an average pixel area of the object over the sequence of images. In some embodiments, the normalization is based on a function that assigns an actual speed to the object based on the average speed of the step <NUM> and the size of the object.

Referring now to <FIG>, a drawing <NUM> illustrating a homography that can be used to translate points from a first image to a second image is shown, according to an exemplary embodiment. In <FIG>, three planes are shown, a first image plane <NUM> (e.g., first camera or first camera position of the first camera), a second image plane <NUM> (e.g., second camera of second camera position of the first camera), and a planar surface <NUM> (e.g., the world plane). The points may exist on the planar surface <NUM> and can be detected on the first image <NUM> and the second image <NUM>.

The planar surface <NUM> can be a surface surveyed by the radar system <NUM> and the points can be radar detections of objects determined by the radar system <NUM>. The first image <NUM> and the second image <NUM> can be two images of the same camera, the camera having moved from a first position to a second position. In some embodiments, the first image <NUM> and the second image <NUM> are images of separate cameras surveying the same scene from different angles. Visual images <NUM> and <NUM> can correspond to the first image <NUM> and the second image <NUM>. The visual image <NUM> and <NUM> illustrate a scene with points, while the points are the same, the points are at different pixel locations based on the angles from which the visual images <NUM> and <NUM> are captured.

A homography, H, may exist between the planar surface and the first image, the planar surface and the second image, and/or between the first image and the second image. The homography may be a matrix of values that can be used to translate points between first image <NUM> and the second image <NUM>. A second homography can translate points between the image <NUM> and the planar surface <NUM>. A third homography can translate points between the second image <NUM> and the planar surface <NUM>. In some embodiments, the first image <NUM> is a first position of a camera at a first time and the second image is a second image <NUM> of the camera at a subsequent time. Therefore, the camera manager <NUM> can be configured to use a homography between the first image <NUM> and the second image <NUM> to translate the location of objects between the first image <NUM> and the second image <NUM> as the camera moves. <FIG> provides another example <NUM> of a homography between multiple planes.

A homography may be defined as an invertible mapping h from P<NUM> to P<NUM> such that three points x<NUM>, x<NUM>, x<NUM> lie on the same line if and only if h(x<NUM>), h(x<NUM>), and h(x<NUM>) do. The theorem for a homography may be stated as: A mapping h: P<NUM> → P<NUM> is a homography if and only if there exists a non-singular 3x3 matrix H such that for any point in P<NUM> represented by a vector X it is true that h(X) = HX. A homography can be defined as: <MAT>.

Referring now to <FIG>, a schematic drawing <NUM> illustrating another homography, H, is shown, according to an exemplary embodiment. In <FIG>, two images <NUM> and <NUM> captured by a camera at two different points in time are shown. A point <NUM> in the image <NUM> can be translated to the frame <NUM>. While a single point is shown to be translated between the frames <NUM> and <NUM>, and entire detection of an object, e.g., the shapes illustrated in the images <NUM> and <NUM> can be translated between the two images via the homography.

Referring now to <FIG>, a process <NUM> is shown for estimating homographies that can be performed by the camera manager <NUM>, according to an exemplary embodiment. The security system manager <NUM> can be configured to perform the process <NUM>. Furthermore, the camera manager <NUM> and/or the calibrator <NUM> can be configured to perform the process <NUM>. However, the process <NUM> is described with reference to the calibrator <NUM>. Multiple homographies can be estimated between consecutive frames of a video sequence captured by the camera <NUM> and/or <NUM> using feature correspondence between the frames. Then, the homographies can be concatenated to establish a set of homographies between each camera frame and a certain camera reference frame, Hpart.

The number of homographies concatenated may be dependent on an image including a point to be translated and a target reference frame. For example, for four images, a first image, a second image, a third image, and a fourth image, there may exist three homographies, a first homograhy between the first image and the second image, a second homograhy between the second image and the third image, and a third homograhy between the third image and the fourth image. To translate from the fourth image to the first image, the first homography, the second homography, and the third homography can be concatenated and used for the translation. To translate from the third image to the first image, only the first and second homographies may be concatenated and used for the translation.

In step <NUM>, the calibrator <NUM> can find features in a first frame and a second frame. In some embodiments, rather than determining features in two frames, the calibrator <NUM> identifies features in more than two frames. The calibrator <NUM> can use an oriented FAST and rotated BRIEF (ORB) algorithm to detect the features. Furthermore, the calibrator <NUM> can use any algorithm to detect the features, e.g., neural networks, decision trees, Bayesian networks, etc..

In step <NUM>, the calibrator <NUM> can determine a correspondence between the features. The calibrator <NUM> can compare the features of the first frame and the features of the second frame to identify whether the features correspond. For example, the features of the first frame may identify a particular object, a vehicle, a stop sign, a building window, etc. The features of the second frame can be compared to the features of the first frame to identify whether the second frame also includes images of the vehicle, stop sign, building window, etc..

In step <NUM>, based on the correspondence between the features of the first frame and the second frame as identified in the step <NUM>, the calibrator <NUM> can find a homography, H, using random sample consensus (RANSAC) which may randomly select features of the first and second frames that correspond to determine the homography. The homography determined with RANSAC, in step <NUM>, can be fine tuned with a mean squared error (MSE) algorithm.

Referring now to <FIG>, an illustration <NUM> of points in a camera plane <NUM> of a camera <NUM> and a world plane <NUM> is shown, according to an exemplary embodiment. A standard calibration may not work well for mapping between the camera plane <NUM> and the world plane <NUM>, e.g., mapping angles to pixels. A standard calibration may not be accurate enough and may fail for an object that is close to a camera. In view of these shortcomings, the calibrator <NUM> is configured to determine a homography between a unit-sphere for the camera and a world plane; this is described in further detail in <FIG>.

Referring now to <FIG>, illustrations <NUM>-<NUM> are shown exemplifying the homography that the calibrator <NUM> is configured to determine to map points between a unit-sphere <NUM> and a world plane <NUM>, according to an exemplary embodiment. The calibrator <NUM> gathers a set of points (e.g., between <NUM>-<NUM> points) to generate the mapping between the unit-sphere <NUM> and the world plane <NUM>. Because the camera <NUM> may rotate, rather than modeling the camera plane as a flat plane, e.g., the camera plane <NUM>, the camera plane is better modeled as the unit-sphere <NUM> since the camera can pan and zoom.

The calibrator <NUM> can be configured to determine the sphere-to-plane homography using non-convex optimization. This may allow the calibrator <NUM> to accurately map between visual tracks of the camera <NUM> and corresponding world locations. The calibrator <NUM> can use the equation below including two terms to determine the sphere-to-plane homography: <MAT> where ∥ ∥b is the geodesics distance of points on the unit sphere, ∥ ∥p is the distance on the plane, A is a three by three matrix of real numbers (e.g., the sphere-to-plane homography), <MAT> is a point in the world plane <NUM>, and <MAT> is a point on the sphere <NUM>. The calibrator <NUM> can be configured to utilize a tensorflow optimization tool to determine values for the sphere-to-plane homography. The values can be selected such that the geodesics distance on the sphere and the plane distance are minimized. The equation above can sum the geodesics distances and plane distances of a set of points. The optimization of the above equation selects a sphere-to-plane homography, A, that minimizes the sum.

The summation of the above equation can be understood, for a particular value of i, corresponding to a particular point ( <MAT>) on the sphere <NUM> and a corresponding point ( <MAT>) on the world plane <NUM>, as a sum of a geodesics distance between the point in the world plane <NUM> translated onto the sphere <NUM> ( <MAT>) and the corresponding point on the sphere <NUM> ( <MAT>). The geodesics distance would optimally be zero, i.e., the homography can translate with no error. However, because the homography is not perfect, the optimization can select values such that error in translation is minimized. The equation further takes into account the translation from the sphere <NUM> to the world plane <NUM>, i.e., the point on the sphere <NUM> ( <MAT>) translated on the sphere <NUM> with an inverse of the homography ( <MAT>) and the corresponding point on the world plane <NUM> ( <MAT>). Again, optimally, the distance would be zero, i.e., the homography can translate with no error.

Referring now to <FIG>, a process <NUM> is shown for determining a sphere-to-plane homography that can be performed by the calibrator <NUM>, according to an exemplary embodiment. The security system manager <NUM> can be configured to perform the process <NUM>. Furthermore, the camera manager <NUM> and/or the calibrator <NUM> can be configured to perform the process <NUM>. While any computing system or device as described herein can be configured to perform he process <NUM>, the process <NUM> is described with reference to the calibrator <NUM>, the camera manager <NUM> and/or the radar system manager <NUM>.

In step <NUM>, the calibrator <NUM> receives a set of points on a camera sphere corresponding to a set of points on a world plane. The set of points on the camera sphere may be pixel coordinates each associated with a world plane radar coordinate. The points of the camera sphere and the world plane can be associated such that a point on the camera sphere is the same point on the world plane, i.e., if a camera views a point with a particular pixel coordinate, the point viewed by the camera has a corresponding world plane coordinate. The correlation between camera and world plane coordinates can be predefined by a user. Furthermore, the correspondence between camera points and world points can be determined via the process of <FIG>, the process of <FIG>, and/or the process of <FIG>.

In step <NUM>, the calibrator <NUM> can perform an optimization to determine a sphere-to-plane homography for translating between points of a camera sphere and a world plane. The calibrator <NUM> can minimize the below equation to identify a 3x3 matrix, A, of real numbers. The optimization may select values for the matrix that are optimal, e.g., values associated with a small or minimal amount of error.

Minimizing the above equation may result in a minimization of a summation of distance errors for all of the points of the step <NUM>. The distance errors may be based on both a geodesics distance on the sphere and a plane distance on the world plane. The geodesics distance may be a distance between two sphere points, an original sphere point and a corresponding point on the world plane translated onto the sphere with the sphere-to-plane homography. Ideally, the geodesics distance would be zero. However, due to error, the geodesics distance will not be zero and thus the optimization attempts to minimize the geodesics distance, thus minimizing the error.

Furthermore, the summation includes a plane distance which is based on the point on the sphere translated onto the plane with the sphere-to-plane homography and the corresponding point on the world plane. Again, ideally, the plane distance would be zero but is not due to error. Thus, the optimization also takes into account error in the sphere-to-plane homography when translating from the sphere to the plane and attempts to minimize the error.

The resulting sphere-to-plane homography of the step <NUM> can be stored by the calibrator <NUM>. Furthermore, the calibrator <NUM> may provide the sphere-to-plane homograph to the camera manager <NUM> for managing and analyzing images received from the security cameras <NUM> and <NUM> and/or the radar system <NUM>. In step <NUM>, the camera manager <NUM> receives a new image with a new point. The image may be received by the camera manager <NUM> from one of the cameras <NUM> and <NUM>. The camera manager <NUM> may identify the point via an image processing algorithm, e.g., the camera manager <NUM> may identify that the new point corresponds to an object of interest, e.g., a person. The camera manager <NUM> can determine a location of the point on the world plane with the sphere-to-plane homography determined in the step <NUM>.

In step <NUM>, the radar system manager <NUM> can receive a detection of a second new point in the world plane. The radar system <NUM> can identify an object but the object may need to be correlated to an object classified by the camera manager <NUM> via images received from a camera. To perform the correlation, radar system manager <NUM> and/or the camera manager <NUM> can use an inverse of the sphere-to-plane homography (e.g., an inverse of the matrix representing the sphere-to-plane homograph).

Referring now to <FIG>, radar tracks <NUM> and camera tracks <NUM> are shown, according to an exemplary embodiment. The camera tracks <NUM> may be object tracks for a camera space (e.g., a screen space) that is based on images captured by cameras <NUM> and/or <NUM>. The radar tracks <NUM> may be object tracks for a radar space (e.g., a world space) captured by the radar system <NUM> (or captured by the cameras <NUM> and/or <NUM> and transformed into the world space). The location of a camera <NUM> which captures the camera tracks <NUM> is shown in the radar tracks <NUM>. Since a radar object detection may be in world coordinates, e.g., latitude and longitude, angle and distance from the radar, etc. it may not directly match up with camera objects which are in pixels (Px, Py) that depend on camera pan, tilt, or zoom.

However, the manager <NUM> can be configured to map between the two spaces so that a detection by radar system <NUM> can be used to control a security camera. For example, the camera manager <NUM> could use the sphere-to-plane homography determined in the process of <FIG> to translate points between the twos planes. The camera may not be calibrated, i.e., internal and/or external parameters may be unknown. The manager <NUM> can make multiple modeling assumptions to determine a correspondence between radar and camera tracks. The manager <NUM> may assume that world geometry is unknown and assumed to be planar and that the camera location is known and is given in radar coordinates (e.g., the camera may include a global positioning system (GPS) on the camera or is provided by an installation technician).

Referring now to <FIG>, a chart <NUM> is shown for illustrating three calibration processes and the resulting radar-camera correspondence accuracy for each, according to an exemplary embodiment. The processes compared in the chart <NUM> illustrate multiple external camera calibration processes. A manual user calibration process <NUM> is shown to require the most calibration effort but produce the highest correspondence accuracy between the radar detections and the camera detections. The manual user calibration process <NUM> is described in greater detail in <FIG>. A semi-automatic calibration process <NUM> is described in greater detail in <FIG> while the fully automatic calibration process <NUM> is described in greater detail in <FIG>. For every zoom level, there may be a correlation between a change in pixels and a pan and/or change tilt in angles. A manual calibration may not be practical for every zoom level.

Referring now to <FIG>, the manual user calibration process <NUM> is shown in greater detail for determining a correspondence between radar detections and camera detections, according to an exemplary embodiment. The security system manager <NUM> can be configured to perform the process <NUM>. Furthermore, the camera manager <NUM> and/or the calibrator <NUM> can be configured to perform the process <NUM>. While any computing device described herein can be configured to perform the process <NUM>, the process <NUM> is described with reference to the calibrator <NUM>.

In step <NUM>, the calibrator <NUM> can receive one or more images of a predefined object with predefined calibration markings for calibrating the camera <NUM> and/or <NUM>. Each of the one or more images can be captured at multiple zoom levels. The image may be image <NUM> as shown in <FIG>. The predefined object of the image <NUM> is a chess board with squares of predefined distances. The size of each square and the distance between each square may be known.

In step <NUM>, the calibrator <NUM> can calculate a projection matrix P for each zoom-level of the camera <NUM> and/or <NUM>. The projection matrix can be determined based on predefined characteristics of the calibration markings, e.g., size, distance between markings, shape, etc. The projection matrix may be a mapping between a two dimensional image point and a three dimensional world point, <MAT> <MAT> where x is the two dimensional world point, P is the projection matrix, and X is the three dimensional world point. The process <NUM> may not be practical for every zoom level. For example, it may be time consuming to perform a calibration for every zoom level. Furthermore, some zooms of cameras may be continuous. In these instances, projection matrices for a predefined representative set of zoom levels can be determined.

In step <NUM>, the calibrator <NUM> can project camera tracks of an object to a world plane with the projection matrix (or matrices if the camera changes zoom) considering the motion abilities of the camera. The motion capabilities can be accounted for by first translating all detections of multiple frames to a first frame via homographies and then translating between the first frame and the world plane. This can account for any movement that the camera performs to track the object.

In step <NUM>, the calibrator <NUM> receive radar detections of the object from the radar system <NUM> and can match between the translated camera detections of the world plane and the detections of the radar system in the world plane. In some embodiments, an association algorithm, e.g., a Hungarian algorithm is used to determine which points of the camera track correspond to which points of the radar track.

Referring now generally to <FIG>, the semi-automatic calibration process <NUM> is shown, according to an exemplary embodiment. With the calibration of <FIG>, a camera scene can be used to semi-automatically calibrate a camera and a radar system via a single object walking in a semi-planar world. The calibrator <NUM> can receive the moving objects location from the radar system <NUM> (e.g., the world space) and can also receive a corresponding detection of the moving object from the camera <NUM> and/or <NUM> (e.g., in the camera space). Based on the locations in the radar world space and the camera detection in the camera world space, the calibrator <NUM> can be configured to find a mapping between the camera space and the world space.

Referring more particularly to <FIG>, the process <NUM> is shown in greater detail, according to an exemplary embodiment. The security system manager <NUM> can be configured to perform the process <NUM>. Furthermore, the camera manager <NUM> and/or the calibrator <NUM> can be configured to perform the process <NUM>. While any computing device described herein can be configured to perform the process <NUM>, the process <NUM> is described with reference to the calibrator <NUM>. The process <NUM> can be performed for the cameras <NUM> and/or <NUM> and radar system <NUM> both tracking a moving object, an example of the tracking data of both a camera and the radar system <NUM> is shown in <FIG>.

In step <NUM>, the calibrator <NUM> can receive a set of images from a camera, the camera moving to track the moving object and can determine homographies between images as the camera moves to track the moving object. The set of images may be frames of a video captured by the camera. The calibrator <NUM> may determine the homographies between the images by analyzing each image to classify the object and then identify the homography according to the location of the classified object in each image. The determination of the homography can be the same as, or similar to, the determination of the homography described with reference to <FIG>.

In step <NUM>, the calibrator <NUM> can transfer all detections to a first image <NUM> (illustrated in <FIG>) via the homographies determined in the step <NUM>. In this regard, the movement of the camera to track the object is compensated for. The homographies can be concatenated to transfer the object from each image of the set of images receive din the step <NUM> to the first image <NUM>.

In step <NUM>, the calibrator <NUM> can determine a homography between the first image <NUM> and the world plane <NUM>, the world being the coordinate system of the radar system <NUM>. In some embodiments, the calibrator <NUM> receives radar data from the radar system <NUM>, the radar data indicating the location in the world plane of the object corresponding to each detection of the object in the frames of the step <NUM>. The calibrator <NUM> can determine the homography as described with reference to <FIG> and/or <FIG>.

In step <NUM>, the calibrator <NUM> can transfer detections using concatenated homographies of the step <NUM> and the homography between the first image <NUM> and the world plane <NUM> of the step <NUM> to the world coordinates. The calibrator <NUM> can first translate points of the set of images to the first image <NUM> with the homographies determined in the step <NUM> and then transfer detections in the first image <NUM> to the world plane <NUM> using the homography of step <NUM>.

In step <NUM>, the calibrator <NUM> can determine the correspondence between received radar tracks in world coordinates. The calibrator <NUM> can determine the correspondence based on the coordinates of the radar system <NUM> and the transferred detections of step <NUM> of the camera. The correspondence can be determined via a matching algorithm, e.g., a Hungarian algorithm. The result of the correspondence can be a pairing between detections of the camera and detections of the radar system <NUM>.

Referring now to <FIG> three images of a camera and detections of a radar system are shown illustrating the calibration process <NUM> of <FIG>, according to an exemplary embodiment. The camera image, images <NUM>, <NUM>, and <NUM> may be an images captured by one of the cameras <NUM> and/or <NUM> over time to track a moving object. The camera manager <NUM> can control each the camera capturing the images shown in images <NUM>, <NUM>, and <NUM> to track the moving object. A world coordinate system of images <NUM>, <NUM>, and <NUM> illustrate detections of the radar system <NUM>. In <FIG>, the open circles indicate the tracking by the camera while the filled circles indicate the tracking by the radar system <NUM>. As shown in <FIG>, the tracking of the radar system <NUM> is transferred into the images <NUM>, <NUM>, and <NUM> and similarly, the detections of the camera are transferred into the world coordinates in the images <NUM>, <NUM>, and <NUM>. The translation can be performed according to the process <NUM> with the homographies determined in the process <NUM>.

Referring again to <FIG>, the x-axis in an image space is shown to be time correlated with an angle in a world space. The relationship can be used in the fully automatic calibration process <NUM> of <FIG>. Referring now to <FIG>, the fully automatic calibration process <NUM> is shown, according to an exemplary embodiment. The process <NUM> can be performed by the calibrator <NUM>. The security system manager <NUM> can be configured to perform the process <NUM>. Furthermore, the camera manager <NUM> and/or the calibrator <NUM> can be configured to perform the process <NUM>. While any computing device described herein can be configured to perform the process <NUM>, the process <NUM> is described with reference to the calibrator <NUM>.

In step <NUM>, the calibrator <NUM> can receive a set of images from a camera and transfer all object detections of a sequence of received image to a first image of the sequence using one or more homographies. In some embodiments, the calibrator <NUM> can determine the homographies and then translate the points to the first image. The step <NUM> may be the same as, or similar to the steps <NUM>-<NUM>.

In step <NUM>, the calibrator <NUM> can represent camera tracks of the detections transferred to the first image as pixel values, the pixel values corresponding to an angle from the object to the camera. In step <NUM>, the calibrator <NUM> can receive radar tracks of the object from the radar system <NUM> and represent the radar track received from the radar system <NUM> as a horizontal angle to the camera. In some embodiments, the radar system <NUM> determines the detections of the radar tracks as coordinate values which, based on a location of the camera, can be translated into horizontal angles to the camera. In some embodiments, the radar system <NUM> determined the track in horizontal angles to the radar system <NUM> and includes the difference in angles between the radar system <NUM> and the camera system in the below equation.

In step <NUM>, the calibrator <NUM> can calculate a distance between the radar tracks of the step <NUM> and the camera tracks of the step <NUM>. The calibrator <NUM> can use the equation below to determine the distance: <MAT> where a is the field of view of the camera divided by the pixel number of the camera and b is the difference between the camera azimuth and the radar azimuth. Radar is the radar angle while Camera is the horizontal pixel number. Accordingly, the above equation can convert the pixel number into a corresponding angle (a * Camera), find the difference in angle between the radar detection and the translated camera detection (Radar - a * Camera) and compensate for an angle difference between the radar system determining the radar tracks and the camera system (-b). The result of the above equation may be the shortest distance between the radar track and the camera track.

In step <NUM>, the calibrator <NUM> can match the camera and radar tracks. The calibrator <NUM> can use a marriage/Hungarian algorithm to perform the track matching which can be based on the distances between the radar tracks and the camera tracks. In some embodiments, the step <NUM> determines a distance between each radar detection of the radar tracks and each camera detection of the camera tracks and uses the matching algorithm to pair the detections of the radar tracks and the camera tracks based on all of the distances.

Referring now to <FIG>, graphs <NUM> and <NUM> are shown illustrating the track matching of the process <NUM> of <FIG>, specifically, determining the distance between the radar tracks and the camera tracks, according to an exemplary embodiment. The graph <NUM> can illustrate the pixel and/or angle tracks of the moving object determined for the camera and/or the radar system in steps <NUM> and <NUM>. The graph <NUM> illustrates the matched tracks. In graphs <NUM>-<NUM>, marker <NUM> represents the radar track in angles while the marker <NUM> represents the camera track in horizontal pixels. The matching can be performed by determining a distance between the radar tracks and camera tracks with the equation above.

Referring now to <FIG>, an example of camera control based on radar data is shown, according to an exemplary embodiment. Current radar to camera setup is time consuming and not accurate enough. The semi-automatic calibration process <NUM> described with reference to <FIG> can be used and/or modified to solve this problem. The homography between camera and world spaces can point the camera to the radar targets. The manager <NUM> can receive and/or determine bounding-box pixel location from the network and center the camera on the object. A homography may exist between the world and the camera but does not have the projection matrix. The manager <NUM> can determine a radar location in the image (Xrad) and can calculate a world location of a center of an image plane <NUM> of the camera, (<NUM>,<NUM>) -> A. The manager <NUM> can determine a camera pan to be the angle between A and B illustrated in <FIG>. The manager <NUM> can determine a tilt to be the angle between Xrad and B.

Referring more particularly to <FIG>, the process <NUM> is shown in greater detail, according to an exemplary embodiment. The security system manager <NUM> can be configured to perform the process <NUM>. Furthermore, the camera manager <NUM> and/or the calibrator <NUM> can be configured to perform the process <NUM>. While any computing device described herein can be configured to perform the process <NUM>, the process <NUM> is described with reference to the camera manager <NUM>.

In step <NUM>, the camera manager <NUM> can receive, from the radar system manager <NUM> and/or the radar system <NUM>, a detection of an object represented in <FIG> as Xrad. Using a homography between a world plane and the image plane <NUM>, the camera manager <NUM> can translate the detection Xrad in the world plane to a location, x, in the image plane <NUM>. The homography can be the homography (or reverse homography) described with reference to <FIG> and/or the homography of <FIG>. In step <NUM>, a center location, a, of the image plane <NUM> can be translated to the world plane via the homography, or the reverse of the homography, used in the step <NUM> to determine a center location, A, in the world plane.

In step <NUM>, the camera manager <NUM> can determine a pan pixel, b, on the image plane <NUM> based on the location of the object in the image, x, and the center of the image plane, a. The camera manager <NUM> can be configured to determine a horizontal distance between a and x and use the resulting horizontal distance as b. The pan pixel, b, may have a corresponding real world location. In step <NUM>, the camera manager <NUM> can translate the pan pixel, b, to a corresponding world plane pan location, B. The translation can be performed with the homography or reverse homography as described with reference to the steps <NUM>-<NUM>.

In order to center the object of corresponding to the location Xrad in the world plane, the camera may be operated according to a camera pan value and a camera tilt value. The camera pan and the camera tilt can be determined in the steps <NUM> and <NUM> as angles, α and β as shown in <FIG>. In step <NUM>, the camera manager <NUM> can determine a the camera pan that would center the object within the camera by determining a pan angle between the center location, A, and the pan location, B. The camera manager <NUM> may store the location of the camera and, the distances from the camera to the locations of A and B may be known and used by the camera manager <NUM> to determine the pan angle, α.

In step <NUM>, the camera manager <NUM> can determine a require camera tilt to center the object in the camera by determining a tilt angle between the radar location of the object, Xrad, and the pan location, B. Again, the camera manager <NUM> may store the location of the camera and, the distances from the camera to the locations of Xrad and B may be known and used by the camera manager <NUM> to determine the tilt angle, β. In step <NUM>, based on the camera pan and the camera tilt determined in the steps <NUM> and <NUM>, the camera manager <NUM> can operate the camera to center the object detected by the radar system manager <NUM> in the step <NUM> within the center of the camera.

Referring now to <FIG>, a process <NUM> is described for controlling camera movement by automatic focal-length estimation, according to an exemplary embodiment. The process described with reference to <FIG> can result in a calibration that can be used to control a camera to zoom on a target and keep it in the center. The process can be a selfcalibrating that does not require human interaction but only a scene with a point of interest. The process may be done once for each camera and/or camera mode and can be an internal calibration.

Referring more particularly to <FIG>, the security system manager <NUM> can be configured to perform the process <NUM>. Furthermore, the camera manager <NUM> and/or the calibrator <NUM> can be configured to perform the process <NUM>. While any computing device described herein can be configured to perform the process <NUM>, the process <NUM> is described with reference to the calibrator <NUM>. For each zoom of a camera, the calibrator <NUM> can perform the steps <NUM>-<NUM> to determine a focal length for each zoom. Once the focal lengths are determined for each zoom, the focal length verse zoom can be fit to a function in step <NUM>.

In step <NUM>, the camera manager <NUM> can pan the camera by a predefined number of degrees, i.e., by β degrees. A previous center of the image, a, may be offset from the center of the image by β degrees as a result of the pan. In step <NUM>, the camera manager <NUM> can estimate the pan distance corresponding to the pan angle with a homography. The pan distance can be a horizontal distance in pixel units, Δpixel. The homography may be predefined and/or can be determined. The homography may be the same as the homography between images as described with reference to <FIG>.

In step <NUM>, the camera can determine the focal length fz based on the pan distance in horizontal pixels determined in the step <NUM> and the pan angle β used to operate the camera in step <NUM>. The focal length, fz, can be the distance in pixels dividend by the tangent of the pan angle β: <MAT>.

The steps <NUM>-<NUM> can be repeated a number of times for each of the zoom lenths resulting in multiple sample focal lengths for each zoom length. The multiple samples of a single zoom length are illustrated in <FIG> as a slope, i.e., Δpixel verse tg(β). The resulting focal length can be the focal length of all sample pairs of Δpixel and tg(β) or can be the slope of a line fit to the set of sample pairs.

Once focal lengths of the camera have been determined for each of multiple zoom distances of the camera, the camera manager <NUM> can perform a global optimization by fitting the focal lengths to a monotonous-increasing function. The calibrator <NUM> can be configured to perform linear programming to fit the values of the focal length to the closest monotonous-increasing function using the equations: <MAT> <MAT> where <MAT> can be an objective function and wi ≤ wi+<NUM> can be a constraint for the minimization of the objective function. fi may represent one focal length of a set of focal lengths, the set of focal lengths being focal lengths for multiple zoom levels. wi may be the corresponding value of the monotonous-increasing function. Optimally, the difference, wi - fi, is zero. However, due to errors, the difference is not zero and thus the optimization minimizes wi - fi such that a monotonous-increasing function, w, is selected with minimal error. The function, w, is forced to a monotonous-increasing function with the constraint wi ≤ wi+<NUM> which causes each value of w to be greater than, or equal to, a previous value. Therefore, the function w never decreases in value.

Once the focal-length is computed for each zoom, the camera manager <NUM> can translate a pixel distance into a degree. The camera manager <NUM> can be configured to use the fit monotonous-increasing function to perform the translation. For example, if the camera manager <NUM> needs to determine a panning or tilting distance, the panning or tilting angle can be determined as the inverse tangent of the pixel distance divided by the focal length. The focal length used in the computation which can be determined with the fit monotonous increasing function and a current zoom level, i.e., the focal length can be the output of the fit monotonous increasing function with an input zoom level. <FIG> provides a representation of the calculation for determining the focal length, representation <NUM>, shown to include a representation of an image plane <NUM>. <FIG> provides a plot <NUM> of the focal lengths and the corresponding zooms fit to a monotonous-increasing function <NUM>.

Referring now to <FIG>, a plot <NUM> illustrating the estimated change in pixels verse the angle of camera movement is shown, according to an exemplary embodiment. The slope of the trend <NUM> of the pixel change verse the tangent of the angle change is the focal length of the camera. As can be seen, the plot <NUM> is noisy, i.e., it includes error. Therefore, the optimization can be performed by the calibrator <NUM>. An example of a fitting via optimization described with reference to <FIG>.

Referring now to <FIG>, another fitting of focal lengths verses zoom to a monotonous increasing function <NUM> is shown by plot <NUM>. Zoom to focal-length translation allows for an accurate movement of a camera to a target. Focal length should increase along with zoom level. However, as can be seen in the plot <NUM>, not all the points follow this trend. Therefore, as previously mentioned, the calibrator <NUM> can be configured to perform linear programming to fit the values of the plot <NUM> to the closest monotonous-increasing function using the equations: <MAT> <MAT> where <MAT> can be an objective function and wi ≤ wi+<NUM> can be a constraint for the minimization of the objective function.

Referring now to <FIG>, a process <NUM> for calibrating camera movement with pixel-information is shown that can be performed by the calibrator <NUM>, according to an exemplary embodiment. The process <NUM> can be an automatic calibration that maps camera rotation directly to a radar space. The process <NUM> can use pan and tilt instead of a homography between sequences of images to translate between a camera plane and a world plane. The process <NUM> may be invariant to the camera intrinsic parameters and/or zoom level. The calibration can be done directly in the world coordinate system. The security system manager <NUM> can be configured to perform the process <NUM>. Furthermore, the camera manager <NUM> and/or the calibrator <NUM> can be configured to perform the process <NUM>. While any computing device described herein can be configured to perform the process <NUM>, the process <NUM> is described with reference to the calibrator <NUM>.

In step <NUM>, for a set of world-points, the calibrator <NUM> can orient a camera to center a view on the target and the calibrator <NUM> can retrieve and/or record a pan and/or tilt value for the camera for each of the movements. In step <NUM>, for each pan and tilt, the calibrator <NUM> can generate a direction ray, a projection onto a virtual screen <NUM>. Each direction ray can include a pan angle and a tilt angle, the pan and tilt angles determined in the step <NUM>.

In step <NUM>, the calibrator <NUM> can determine a homography between the virtual screen <NUM> and the world plane <NUM>. The calibrator <NUM> may receive radar data of the radar system that indicates a location of each point on the world plane <NUM> and thus the homography and between the virtual screen <NUM> and the world plane <NUM> can be determined based on a correspondence between each direction ray (pan and tilt angle) and the world plane coordinates.

In step <NUM>, using the homography determined in the step <NUM>, the calibrator <NUM> can translate between one or more world coordinates and the pan and tilt of the camera. Similarly, in step <NUM>, the calibrator <NUM> can use a reverse homography of the homography determined in the step <NUM> to translate between the pan and tilt settings to the world coordinates.

Referring now to <FIG>, images <NUM>-<NUM> that can be captured by one of the cameras <NUM> and/or <NUM> are shown, according to an exemplary embodiment. The images <NUM>-<NUM> illustrate tracking and zooming in on an individual that can be performed by the camera manager <NUM>. More pixels can improve the confidence of object detection performed by the camera manager <NUM>; therefore, the camera capturing the images <NUM>-<NUM> may be a high resolution camera. The camera manager <NUM> can be configured to control the camera so that the individual is kept in the center of the image. The control and tracking may be based on a Kalman filter, a matching algorithm, and a homography as described with reference to <FIG>. A new location of the individual can be determined and/or transmitted to a monitoring system every second. While the camera is zoomed on target the system may also monitor other objects in the frame.

The camera manager <NUM> can generate the bounding box <NUM> and cause the bounding box <NUM> to be included within the images <NUM>-<NUM>. The bounding box can include information generated by the camera manager <NUM>. For example, the classification determined by the camera manager <NUM> can be included within the bounding box <NUM>, i.e., "person. " Furthermore, the camera manager <NUM> can cause the bounding box to include a score associated with the classification, e.g., how likely the object captured within the frames <NUM>-<NUM>, "<NUM>. " The camera manager <NUM> can cause the bounding box <NUM> to include a speed of the person as well. The camera manager <NUM> can determine the speed of the individual with a Kalman filter.

Referring now to <FIG>, images <NUM>-<NUM> illustrating detecting, tracking, classifying objects by the camera manager <NUM>. The camera capturing the images <NUM>-<NUM> may be the camera <NUM> and/or <NUM>. As can be seen, the camera is moving forward. It is seen that the camera continues to properly detect, track, and classify the objects. Furthermore, the camera manager <NUM> can reliably detect, track, and/or classify the objects in real time as the camera moves. The performance of the camera manager <NUM> can reduce false detections. No false detections are shown in images <NUM>-<NUM> even though the camera is moving.

Referring now to <FIG>, images <NUM>-<NUM> are shown illustrating detection and tracking of a user in a camera view and a world view, according to an exemplary embodiment. For the images <NUM>-<NUM>, the left side indicates a camera view, e.g., a camera captured image via one of the cameras <NUM> and/or <NUM>. The right side indicates a world view. The world view can indicate radar coordinates. In some embodiments, the camera capturing the left side of images <NUM>-<NUM> can be controlled capture images of the people if the radar system <NUM> detects the people. In some embodiments, the camera can identify the people and determine the locations in the world view of the individuals solely based on the camera detected images and not based on the radar system <NUM>. In this regard, the indications of the individuals on the right side of images <NUM>-<NUM> can be determined based on either the radar system <NUM> and/or the images of the cameras <NUM>-<NUM>.

Referring now to <FIG>, images <NUM>-<NUM> and <NUM>-<NUM> are shown indicating detection, tracking, and classification of multiple individuals by the camera manager <NUM>, according to an exemplary embodiment. In images <NUM>-<NUM>, frames captured by the camera <NUM> and/or <NUM> are shown. As can be seen, the camera is operated to keep all three of the moving individuals within the frame of the camera. In images <NUM>-<NUM>, the locations of the individuals are shown in world coordinates. The locations shown in the images <NUM>-<NUM> can be detections of the camera translated to world coordinates. Furthermore, the locations can be detections via the radar system <NUM>. Each of the objects (people) in <FIG> are shown to be bounded by a box. The box may include an indication including an identification (e.g., person, animal, car, boat, plane, motorcycle, etc.), a score, and a speed of the object. The score may indicate the certainty of the classification of the object.

In some embodiments, a user can provide an input to one of the images <NUM>-<NUM> via a user device. For example, the user may click on a location within the images <NUM>-<NUM>. The location on which the user clicks may correspond to a world-space location. Based on a homography between a camera and the world space, the camera system can orient the camera to view the location clicked-on by the user. For example, the sphere-to-plane homograhy of <FIG> (or the other homography and control algorithms discussed herein) can be used to orient the camera according to the input by the user.

The present disclosure contemplates methods, systems and program products on any machine-readable media for accomplishing various operations. The embodiments of the present disclosure may be implemented using existing computer processors, or by a special purpose computer processor for an appropriate system, incorporated for this or another purpose, or by a hardwired system. Embodiments within the scope of the present disclosure include program products comprising machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such machine-readable media can be any available media that can be accessed by a general purpose or special purpose computer or other machine with a processor. By way of example, such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code in the form of machine-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer or other machine with a processor. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a machine, the machine properly views the connection as a machine-readable medium. Thus, any such connection is properly termed a machine-readable medium. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions.

Claim 1:
A building radar-camera system (<NUM>), comprising:
- one or more memory devices (<NUM>) storing instructions thereon that, when executed by one or more processors (<NUM>), cause the one or more processors (<NUM>) to:
- receive one or more images from a camera (<NUM>, <NUM>), the one or more images comprising first locations within the one or more images of one or more points on a world-plane (<NUM>);
- receive radar data from a radar system (<NUM>), the radar data indicating second locations on the world-plane (<NUM>) of the one or more points;
- receive a correspondence between the first locations and the second locations of the one or more points, the correspondence associating each of the first locations with one of the second locations; and
- generate a sphere-to-plane homography, the sphere-to-plane homography translating between points captured by the camera (<NUM>, <NUM>) modeled on a unit-sphere (<NUM>) and the world-plane (<NUM>) based on the correspondence between the first locations and the second locations, by performing an optimization to identify values for the sphere-to-plane homography that minimize one or more error values.