Historical gaze heat map for a video stream

A method and a video management system is disclosed. The method may include receiving a video stream from a camera and displaying the video stream on a display. The method may include obtaining, via an eye tracking sensor, gaze information for an operator watching the display. The method may include generating a historical gaze heat map for the video stream for a time period based on the obtained gaze information and determining a low interest area for the video stream based on the generated historical gaze heat map. The method may include instructing the camera to decrease a bit rate of the video stream in the low interest area.

BACKGROUND

A monitoring device, such as a camera, may be installed to capture images or video of an area of interest. An operator, or a group of operators, may monitor images or video from the camera on a display that is located remotely from the camera. The data captured by the camera may be processed and sent over a network connection to the display. For example, the images or video may be compressed in order to reduce the amount of data that has to be transmitted across a network. A system may include a large number of cameras transmitting a large amount of data across a network. The large amount of data may tax the resources of the network.

SUMMARY

According to one aspect, a method, performed by a computer device, may include receiving a video stream from a camera; displaying the video stream on a display; obtaining, via an eye tracking sensor, gaze information for an operator watching the display; generating a historical gaze heat map for the video stream for a time period based on the obtained gaze information; determining a low interest area for the video stream based on the generated historical gaze heat map; and instructing the camera to decrease a bit rate of the video stream in the low interest area.

Additionally, the method may include determining a high interest area of the video stream based on the generated historical gaze heat map; and instructing the camera to increase a bit rate of the video stream in the high interest area.

Additionally, the time period may be longer than a day.

Additionally, the time period may correspond to a particular time of a day or a particular day of week, and determining the low interest area of the video stream based on the generated historical gaze heat map may include determining a first low interest area for a first time of day or day of week based on a historical gaze heat map generated for the first time of day or day of week over multiple instances of the first time of day or day of week; and determining a second low interest area for a second time of day or day of week, wherein the second low interest area is different from the first low interest area based on a historical gaze heat map generated for the second time of day or day of week over multiple instances of the second time of day or day of week.

Additionally, determining a low interest area of the video stream based on the generated historical gaze heat map may include identifying a pan, zoom, tilt, rotation, or image type setting for the camera; selecting a subset of the generated historical gaze heat map associated with the identified pan, zoom, tilt, rotation, or image type setting for the camera; and determining the low interest area of the video stream based on the selected subset of the generated historical gaze heat map.

Additionally, the video stream may include a plurality of video streams and wherein the display includes a plurality of displays.

Additionally, the method may include associating an event type with a particular change in the historical gaze heat map; detecting an event of the associated event type; and changing the low interest area based on the particular change in the historical gaze heat map, in response to detecting the event.

Additionally, instructing the camera to decrease the bit rate of the video stream in the low interest area may include instructing the camera to reduce a sampling rate for a sensor associated with the low interest area.

Additionally, instructing the camera to decrease the bit rate of the video stream in the low interest area may include instructing the camera to increase a noise reduction process for the low interest area prior to encoding the video stream.

Additionally, instructing the camera to decrease the bit rate of the video stream in the low interest area may include instructing the camera to adjust an encoding parameter for an encoding processing unit associated with the low interest area.

According to another aspect, a computer system may include a memory to store instructions; and a receiver to receive a video stream from a camera; a display to display the video stream; a processor configured to execute the instructions to obtain, via an eye tracking sensor, gaze information for an operator watching the display, generate a historical gaze heat map for the video stream for a time period based on the obtained gaze information, determine a low interest area for the video stream based on the generated historical gaze heat map, and determine to instruct the camera to decrease a bit rate of the video stream in the low interest area; and a transmitter to send an instruction to the camera to decrease the bit rate of the video stream in the low interest area.

Additionally, the processor may be further configured to determine a high interest area of the video stream based on the generated historical gaze heat map; and the processor may be configured to determine to instruct the camera to increase a bit rate of the video stream in the high interest area, and transmitter may be configured to send an instruction to the camera to increase the bit rate of the video stream in the high interest area.

Additionally, the time period may be longer than a day.

Additionally, the time period may correspond to a particular time of a day or a particular day of week, and the processor may be further configured to determine a first low interest area for a first time of day or day of week based on a historical gaze heat map generated for the first time of day or day of week over multiple instances of the first time of day or day of week; and determine a second low interest area for a second time of day or day of week, wherein the second low interest area is different from the first low interest area based on a historical gaze heat map generated for the second time of day or day of week over multiple instances of the second time of day or day of week.

Additionally, the processor may be further configured to identify a pan, zoom, tilt, rotation, or image type setting for the camera; select a subset of the generated historical gaze heat map associated with the identified pan, zoom, tilt, rotation, or image type setting for the camera; and determine the low interest area of the video stream based on the selected subset of the generated historical gaze heat map.

Additionally, the video stream may include a plurality of video streams and the display may include a plurality of displays.

Additionally, the processor may be further configured to associate an event type with a particular change in the historical gaze heat map; detect an event of the associated event type; and change the low interest area based on the particular change in the historical gaze heat map, in response to detecting the event.

Additionally, the processor may be further configured to determine to instruct the camera to reduce a sampling rate for a sensor associated with the low interest area; and the instruction to decrease the bit rate of the video stream in the low interest area may include an instruction to the camera to reduce the sampling rate for the sensor associated with the low interest area.

Additionally, the processor may be further configured to determine to instruct the camera to increase a noise reduction process for the low interest area prior to encoding the video stream; and the instruction to decrease the bit rate of the video stream in the low interest area may include an instruction to increase the noise reduction process for the low interest area prior to encoding the video stream.

Additionally, the processor may be further configured to determine to instruct the camera to adjust an encoding parameter for an encoding processing unit associated with the low interest area, and the instruction to decrease the bit rate of the video stream in the low interest area may include an instruction to adjust the encoding parameter for the encoding processing unit associated with the low interest area.

DETAILED DESCRIPTION

Implementations described herein relate to a gaze heat map. A monitoring camera streams video of an area of interest to a display. Video streams may have high bandwidth requirements and may consume significant network bandwidth. Furthermore, processing a video stream may require processor and memory resources. A person watching the display, referred to herein as an “operator” or “user,” may find particular areas of a scene shown on the display to be of high interest and may find other areas of the scene to be of low interest. For example, if the camera is monitoring an area with a path and a door, the operator may spend a significant amount of time watching the door and the path, and may spend a relatively low amount of time watching the wall around the door.

An eye tracker may be used to identify the operator's gaze area and therefore to identify a portion of the display, or one or more displays in a group of displays, at which the operator is looking. Over a period of time, a historical gaze heat map may be generated that indicates the amount of time that the operator spends looking at particular locations on a display showing a video stream from a particular camera over a particular time period. The time period may be selected based on an expected variability in the video stream. In some implementations, the time period may be at least one day. In other implementations, the time period may be longer than a day (e.g., one week, one month, etc.).

The historical gaze heat map may, for example, assign a value to each pixel, or set of pixels, in a frame of a video stream, with the assigned value representing the length of time for which the operator's gaze point corresponded to the pixel, or set of pixels. For example, a higher assigned value to a pixel, or set of pixels, may correspond to a longer amount of time spent by the operator looking at the location of the video stream frame corresponding to the pixel, or set of pixels.

The historical gaze heat map may be used to determine a low interest area for the video stream and the camera may be instructed to decrease a bit rate of the video stream in the low interest area. Reducing the bit rate of a video stream in a low interest area of the video stream may result in the technical effect of conserving network resources and reducing processor and memory load for a monitoring system of cameras and monitoring stations. In some implementations, the video stream may include multiple video streams and the display may include multiple displays.

Furthermore, in some implementations, a high interest area for the video stream may be determined based on the historical gaze heat map and the camera may be instructed to increase a bit rate of the video stream in the high interest area.

The historical gaze heat map may store additional information for particular data points and the additional information may be used to refine the gaze heat map with respect to a particular parameter. As an example, for each location (e.g., pixel, set of pixels, etc.) of the video stream in the gaze heat map, the additional information may include information identifying a particular time of day or a particular day of week when the gaze information was collected. Thus, gaze heat maps for different times of day or days of the week may be retrieved from the historical gaze heat map based on multiple instances of gaze information being collected for the video stream for a particular time of day or day of week. Thus, for example, a first low interest area may be determined for a first time of day or day of week based on a historical gaze heat map generated for the first time of day or day of week over multiple instances of the first time of day or day of week, and a second low interest area may be generated for a second time of day or day of week, based on a historical gaze heat map generated for the second time of day or day of week over multiple instances of the second time of day or day of week.

As another example, for each location of the video stream in the gaze heat map, the additional information may include information identifying a particular camera setting when the gaze information was collected. Thus, gaze heat maps for different camera settings may be retrieved from the historical gaze heat map. For example, a pan, zoom, tilt, rotation, or image type (e.g., normal vs. wide angle, etc.) setting for the camera may be selected, a subset of the historical gaze heat map may be generated based on the camera setting selection, and a low interest area of the video stream for the selected camera setting may be determined based on the gaze heat map subset.

As yet another example, for each location of the video stream in the gaze heat map, the additional information may include information identifying a particular event type. An event type may be identified based on a generated alarm, based on an event recorded in a calendar associated with a monitored area and managed by a video management system, based on manual information entered by the operator, and/or based on another technique of identifying an event. An event type may be associated with a particular change in the historical gaze heat map. If the event type is detected in the future, the low interest area may be changed based on the associated change in the historical gaze heat map. For example, a door sensor may indicate that a door has been opened and the operator's gaze may change to an area associated with the door within a particular time of the door sensor being activated. After an association with the door sensor is made, if the door sensor is activated in the future, a low interest area associated with the door may be changed to a high interest area and a bit rate for the door are may be increased.

The bit rate may be reduced at any of a number of points along a processing path from the point of capture of the video data by a sensor array to transmitting an encoded video stream to the display via a network connection. As an example, instructing the camera to decrease the bit rate of the video stream in the low interest area may include instructing the camera to reduce a sampling rate for a sensor associated with the low interest area. As another example, instructing the camera to decrease the bit rate of the video stream in the low interest area may include instructing the camera to lower resolution for the low interest area prior to encoding the video stream. As yet another example, instructing the camera to decrease the bit rate of the video stream in the low interest area may include instructing the camera to increase a noise reduction process for the low interest area prior to encoding the video stream. As yet another example, instructing the camera to decrease the bit rate of the video stream in the low interest area may include instructing the camera to increase a compression parameter value for an encoding processing unit associated with the low interest area.

FIG. 1is a block diagram illustrating an exemplary environment100in an embodiment. Environment100may be, for example, a monitoring system to secure an area or provide public safety. As shown inFIG. 1, environment100may include cameras110-1through110-M, network120, monitoring stations125-1through125-N, displays130-1through130-N, eye trackers140-1through140-N, and/or a video management system (VMS)150.

Cameras110-1through110-M (individually “camera110,” or plural “cameras110”) capture images and/or video of monitored areas106. A monitored area106may be monitored by one or more cameras110. For example, two cameras can monitor area106-1, which includes an object102-1. Objects102may include any object, such as a door, a person, an animal, a vehicle, a license plate on a vehicle, etc.

Camera110may capture image data using visible light, infrared light, and/or other non-visible electromagnetic radiation (e.g., ultraviolet light, far infrared light, terahertz radiation, microwave radiation, etc.). Camera110may include a thermal camera and/or a radar device for radar imaging. The captured image data may include a continuous image sequence (e.g., video), a limited image sequence, still images, and/or a combination thereof. Camera110may include a digital camera for capturing and digitizing images and/or an analog camera for capturing images and storing image data in an analog format.

Camera110may include sensors that generate data arranged in one or more two-dimensional array(s) (e.g., image data or video data). As used herein, “video data” and “video” may be referred to more generally as “image data” and “image,” respectively. Thus, “image data” or an “image” is meant to include “video data” and “videos” unless stated otherwise. Likewise, “video data” or a “video” may include a still image unless stated otherwise. Furthermore, in some implementations, “video data” may include audio data.

Monitoring stations125-1through125-N may include computer devices that are clients of VMS150and that are coupled to displays130-1through130-N (individually “monitoring station125” and “display130,” respectively). In an embodiment, monitoring stations125-1through125-N are also coupled to eye trackers140-1through140-N (individually “eye tracker140”). Monitoring station125and display130enable operators (not shown inFIG. 1) to view images from cameras110. Eye tracker140tracks the gaze of an operator viewing display130. Each monitoring station125, display130, and eye tracker140may be a “client” for an operator to interact with the monitoring system shown in environment100.

Display130receives and displays video stream(s) from one or more cameras110. A single display130may show images from a single camera110or from multiple cameras110(e.g., in multiple frames or windows on display130). A single display130may also show images from a single camera but in different frames. That is, a single camera may include a wide-angle or fisheye lens, for example, and provide images of multiple areas106. Images from the different areas106may be separated and shown on display130separately in different windows and/or frames. Display130may include a liquid-crystal display (LCD), a light-emitting diode (LED) display, an organic LED (OLED) display, a cathode ray tube (CRT) display, a plasma display, a laser video display, an electrophoretic display, a quantum-dot display, a video projector, and/or any other type of display device.

Eye tracker140includes a sensor (e.g., a camera) that enables VMS150(or any other device in environment100) to determine where the eyes of an operator are focused. For example, a set of near-infrared light beams may be directed at an operator's eyes, causing reflections in the operator's corneas. The reflections may be tracked by a camera included in eye tracker140to determine the operator's gaze area. The gaze area may include a gaze point and an area of foveal focus. For example, an operator may sit in front of display130of monitoring station125. Eye tracker140determines which portion of display130the operator is focusing on. Each display130may be associated with a single eye tracker140. Alternatively, an eye tracker140may correspond to multiple displays130. In this case, eye tracker140may determine which display and/or which portion of that display130the operator is focusing on.

Eye tracker140may also determine the presence, a level of attention, focus, drowsiness, consciousness, and/or other states of a user. Eye tracker140may also determine the identity of a user. The information from eye tracker140can be used to gain insights into operator behavior over time or determine the operator's current state. In some implementations, display130and eye tracker140may be implemented in a virtual reality (VR) headset worn by an operator. The operator may perform a virtual inspection of area106using one or more cameras110as input into the VR headset.

Network120may include one or more circuit-switched networks and/or packet-switched networks. For example, network120may include a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a Public Switched Telephone Network (PSTN), an ad hoc network, an intranet, the Internet, a fiber optic-based network, a wireless network, and/or a combination of these or other types of networks.

VMS150may include one or more computer devices, such as, for example, server devices, which coordinate operation of cameras110, display devices130, and/or eye tracking system140. VMS150may receive and store image data from cameras110. VMS150may also provide a user interface for operators of monitoring stations125to view image data stored in VMS150or image data streamed from cameras110. VMS150may include a rule engine to conserve system resources by instructing cameras110to reduce a bit rate for a region that is outside the operator's gaze area.

In some embodiments, environment100does not include a separate VMS150. Instead, the services provided by VMS150are provided by monitoring stations125(e.g., computer devices associated with displays130) and/or cameras110themselves or in a distributed manner among the devices in environment100. For example, cameras110may include a rule engine to conserve system resources by instructing cameras110to reduce a bit rate for a region that is outside the operator's gaze area. Similarly, VMS150may perform operations described as performed by camera110.

AlthoughFIG. 1shows exemplary components of environment100, in other implementations, environment100may include fewer components, different components, differently arranged components, or additional components than depicted inFIG. 1. Additionally or alternatively, any one device (or any group of devices) may perform functions described as performed by one or more other devices.

FIG. 2is a block diagram illustrating exemplary components of a camera110in an embodiment. As shown inFIG. 2, camera110may include an optics chain210, a sensor array220, a bus225, an image processor230, a controller240, a memory245, a video encoder250, and/or a communication interface260. In an embodiment, camera110may include one or more motor controllers270(e.g., three) and one or more motors272(e.g., three) for panning, tilting, rotating, and/or zooming camera110.

Optics chain210includes an enclosure that directs incident radiation (e.g., light, visible light, infrared waves, millimeter waves, etc.) to a sensor array220to capture an image based on the incident radiation. Optics chain210includes one or more lenses212collect and focus the incident radiation from a monitored area onto sensor array220.

Sensor array220may include an array of sensors for registering, sensing, and measuring radiation (e.g., light) incident or falling onto sensor array220. The radiation may be in the visible light wavelength range, the infrared wavelength range, or other wavelength ranges.

Sensor array220may include, for example, a charged coupled device (CCD) array and/or an active pixel array (e.g., a complementary metal-oxide-semiconductor (CMOS) sensor array). Sensor array220may also include a microbolometer (e.g., when camera110includes a thermal camera or detector).

Sensor array220outputs data that is indicative of (e.g., describes properties or characteristics) the radiation (e.g., light) incident on sensor array220. For example, the data output from sensor array220may include information such as the intensity of light (e.g., luminance), color, etc., incident on one or more pixels in sensor array220. The light incident on sensor array220may be an “image” in that the light may be focused as a result of lenses in optics chain210. In some implementations, controller240may reduce the bit rate associated with a particular region of sensor array220by turning off, and/or reduce a sampling rate, of a particular sensor, or a set of sensors, of sensor array220.

Sensor array220can be considered an “image sensor” because it senses images falling on sensor array220. As the term is used herein, an “image” includes the data indicative of the radiation (e.g., describing the properties or characteristics of the light) incident on sensor array220. Accordingly, the term “image” may also be used to mean “image sensor data” or any data or data set describing an image. Further, a “pixel” may mean any region or area of sensor array220for which measurement(s) of radiation are taken (e.g., measurements that are indicative of the light incident on sensor array220). A pixel may correspond to one or more (or less than one) sensor(s) in sensor array220. In alternative embodiments, sensor array220may be a linear array that may use scanning hardware (e.g., a rotating mirror) to form images, or a non-array sensor which may rely upon image processor230and/or controller240to produce image sensor data.

Bus225includes a communication path that enables components in camera110to communicate with each other. Controller240and/or image processor230perform signal processing operations on image data captured by sensor array220. For example, image processor230may perform image processing on images captured by sensor array220, such as noise reduction, filtering, scaling, etc. Controller240may control the operation of camera110and may provide instructions to other components of camera110, such as sensor array220, image processor230, video encoder250, communication interface260, and/or motor controller(s)270.

Controller240and/or image processor230may include any type of single-core or multi-core processor, microprocessor, latch-based processor, and/or processing logic (or families of processors, microprocessors, and/or processing logics) that interpret and execute instructions. Controller240and/or image processor230may include or be coupled to a hardware accelerator, such as a graphics processing unit (GPU), a general purpose graphics processing unit (GPGPU), a Cell, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and/or another type of integrated circuit or processing logic.

Controller240may also determine and control the desired focus and position (e.g., tilt, pan, rotation, zoom, etc.) of camera110. To do so, controller240sends commands to one or more motor controllers270to drive one or more motors272to tilt, pan, rotate, and/or zoom camera110or optically zoom lenses212.

Memory245may include any type of volatile and/or non-volatile storage device that stores information and/or instructions. Memory245may include a random access memory (RAM) or any type of dynamic storage device, a read-only memory (ROM) device or any type of static storage device, a magnetic or optical recording memory device and its corresponding drive, or a removable memory device. Memory245may store information and instructions (e.g., applications and/or an operating system) and data (e.g., application data) for use by processor camera110. Memory245may store information identifying one or more bit rate reduction factors and/or particular sensor array capture, image processing, and/or encoding processes and/or parameters to which the one or more bit rate reduction factors are to be applied.

Memory245may store instructions for execution by controller240, image processor230, video encoder250, and/or communication interface260. The software instructions may be read into memory245from another computer-readable medium or from another device. The software instructions may cause controller240, image processor230, video encoder250, and/or communication interface260to perform processes described herein. For example, camera110may perform operations relating to the image processing (e.g., encoding, noise reduction, transcoding, detecting objects, etc.) in response to controller240, image processor230, and/or video encoder250executing software instructions stored in memory245. Alternatively, hardwired circuitry (e.g., logic) may be used in place of, or in combination with, software instructions to implement processes described herein.

Video encoder250may compress video data based on one or more video codecs, such as an H.262/Moving Pictures Experts Group (MPEG)-2 codec, an H.263/MPEG-2 Part 2 codec, an H.264/MPEG-4 codec, an H.265/MPEG-H High Efficiency Video Coding (HVEC) codec, and/or another type of codec.

Communication interface260includes circuitry and logic circuitry that includes input and/or output ports, input and/or output systems, and/or other input and output components that facilitate the transmission of data to another device. For example, communication interface340may include a network interface card (e.g., Ethernet card) for wired communications or a wireless network interface (e.g., a Long Term Evolution (LTE), WiFi, Bluetooth, etc.) card for wireless communications.

AlthoughFIG. 2shows exemplary components of camera110, in other implementations, camera110may include fewer components, different components, differently arranged components, or additional components than depicted inFIG. 2. Additionally or alternatively, one or more components of camera110may perform functions described as performed by one or more other components of camera110. For example, controller240may perform functions described as performed by image processor230and vice versa. Alternatively or additionally, camera110may include a computing module as described below with respect toFIG. 3.

FIG. 3is a block diagram illustrating exemplary components of a computing module300in an embodiment. Devices such as VMS150, eye-tracking system140, monitoring stations125, and/or display devices130may include one or more computing modules300. As shown inFIG. 3, computing module300may include a bus310, a processor320, a memory330, and/or a communication interface360. In some embodiments, computing module300may also include an input device340and/or an output device350.

Bus310includes a path that permits communication among the components of computing module300or other devices. Processor320may include any type of single-core processor, multi-core processor, microprocessor, latch-based processor, and/or processing logic (or families of processors, microprocessors, and/or processing logics) that interprets and executes instructions. Processor320may include an application-specific integrated circuit (ASIC), an FPGA, and/or another type of integrated circuit or processing logic. Processor320may include or be coupled to a hardware accelerator, such as a GPU, a GPGPU, a Cell, a FPGA, an ASIC, and/or another type of integrated circuit or processing logic.

Memory330may include any type of dynamic storage device that may store information and/or instructions, for execution by processor320, and/or any type of non-volatile storage device that may store information for use by processor320. For example, memory330may include a RAM or another type of dynamic storage device, a ROM device or another type of static storage device, a magnetic and/or optical recording memory device and its corresponding drive (e.g., a hard disk drive, optical drive, etc.), and/or a removable form of memory, such as a flash memory.

Memory330may store instructions for execution by processor320. The software instructions may be read into memory330from another computer-readable medium or from another device. The software instructions may cause processor320to perform processes described herein. Alternatively, hardwired circuitry (e.g., logic) may be used in place of, or in combination with, software instructions to implement processes described herein.

The operating system may include software instructions for managing hardware and software resources of computing module300. For example, the operating system may include Linux, Windows, OS X, Android, an embedded operating system, etc. Applications and application data may provide network services or include applications, depending on the device in which the particular computing module300is found.

Communication interface360may include a transmitter and/or receiver (e.g., a transceiver) that enables computing module300to communicate with other components, devices, and/or systems. Communication interface360may communicate via wireless communications (e.g., radio frequency, infrared, and/or visual optics, etc.), wired communications (e.g., conductive wire, twisted pair cable, coaxial cable, transmission line, fiber optic cable, and/or waveguide, etc.), or a combination thereof. Communication interface360may include a transceiver that converts baseband signals to radio frequency (RF) signals or vice versa and may include an antenna assembly.

In some implementations, computing module300may also include input device340and output device350. Input device340may enable a user to input information into computing module300. Input device370may include, for example, a keyboard, a mouse, a pen, a microphone, a remote control, an audio capture device, an image and/or video capture device, a touch-screen display, and/or another type of input device.

Output device350may output information to the user. Output device350may include a display, a printer, a speaker, and/or another type of output device. Input device340and output device350may enable a user interact with applications executed by computing module300. In the case of a “headless” device (such as a deployed remote camera), input and output is primarily through communication interface360rather than input device340and output device350.

As described in detail below, computing module300may perform certain operations relating to bit rate adjustments based on a historical gaze heat map. Computing module300may perform these operations in response to processor320executing software instructions contained in a computer-readable medium, such as memory330. A computer-readable medium may be defined as a non-transitory memory device. A memory device may be implemented within a single physical memory device or spread across multiple physical memory devices. The software instructions may be read into memory330from another computer-readable medium or from another device. The software instructions contained in memory330may cause processor320to perform processes described herein. Alternatively, hardwired circuitry may be used in place of, or in combination with, software instructions to implement processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.

Computing module300may include other components (not shown) that aid in receiving, transmitting, and/or processing data. Moreover, other configurations of components in computing module300are possible. In other implementations, computing module300may include fewer components, different components, additional components, or differently arranged components than depicted inFIG. 3. Additionally or alternatively, one or more components of computing module300may perform one or more tasks described as being performed by one or more other components of computing module300.

FIG. 4illustrates an exemplary environment400of an operator402viewing display130having eye tracker140in an embodiment. Display130may include any type of display for showing information to operator402. Operator402views display130and can interact with VMS150via an application running on monitoring station125. For example, operator402may watch a video of area106.

Eye tracker140includes a sensor (e.g., a camera) that enables monitoring station125to determine where the eyes of operator402are focused. InFIG. 4, for example, operator402sits in front of display130and the sensor in eye tracker140senses the eyes of operator402. For example, eye tracker140may determine a gaze point410, which may be represented as a location (e.g. pixel values associated with one or more pixels) on display130. Based on the relative position of the operator and the display130, a foveal vision area420(or “area420”) corresponding to the foveal vision of operator402may be estimated. Foveal vision corresponds to the detailed visual perception of the eye, and approximately subtends 1-2 spherical degrees. Accordingly, area420on display130may be calculated and understood to correspond to the part of operator's402vision with full visual acuity.

In an alternative embodiment, foveal vision area420may be determined experimentally during a setup procedure for a particular operator402. Foveal vision area420is in contrast to peripheral vision area430outside of foveal vision area420, which corresponds to the peripheral vision of operator402. Gaze point410is approximately in the center of area420and corresponds to the line-of-sight from gaze point410to the eyes of operator402. In an embodiment, information identifying gaze point410may be transmitted to VMS150.

FIG. 5Aillustrates display130from the perspective of operator402. As shown inFIG. 5A, display130includes gaze point410, foveal vision area420, and peripheral vision area430. Display130also includes a video frame520in which a video stream is presented to operator402. In this example, frame520shows a video stream from camera110of area106, which happens to include a door and an individual who appears to be moving. Operator's402foveal vision area420encompasses the individual and gaze point410is directly on the individual's face. The door displayed in frame520, on the other hand, appears in operator's402peripheral vision area430.

In some implementations, the gaze heat map may be generated based on gaze point410. In other implementations, the gaze heat map may be generated based on foveal vision area420. In yet other implementations, the gaze heat map may be generated based on an area in size somewhere between the size of gaze point410and the size of foveal vision area420. In yet other implementations, the gaze heat map may be generated based on an area that is larger, and centered upon, foveal vision area420.

FIG. 5Balso illustrates display130from the perspective of operator402. In contrast toFIG. 5A, however, display130inFIG. 5Bshows numerous frames520-1through520-N (individually “frame520”; or plural “frame520”). Each frame520-1through520-N may present a different video stream so operator402can monitor more than one area. The different streams may be produced by different cameras110-1through110-M. In other embodiments, each frame520-1through520-N may be displayed on a different display130arranged in front of the operator (e.g., on a wall, in an arc in front of the operator, etc.). Alternatively or additionally, each frame520-1through520-N may display different streams generated by a common camera110-x. For example, camera110-xmay use a “fisheye” lens and capture video from an extended angular area. The video may be processed to reduce distortions introduced by the fisheye lens, and separate the extended angular area into separate video streams corresponding to different areas, which may be separately presented in frames520-1through520-N. As withFIG. 5A, display130inFIG. 5Bincludes gaze point410, foveal vision area420, and peripheral vision area430.

In this example, frame520-1may show a video stream from camera110-1of area106-1; video frame520-2may show a video stream from camera110-2of area106-2; etc. Operator's402foveal vision area420inFIG. 5Bencompasses the majority of frame520-1and gaze point410is close to the individual's face. The door displayed in frame520is also in foveal vision area420. The other frames520-2through520-N, on the other hand, are in operator's402peripheral vision area430. The location of gaze point410and/or foveal vision area420may be used to select and/or designate a particular frame520-xfor subsequent processing which may be different from other frames520. For example, as shown inFIG. 5B, gaze point410may be used to indicate that frame520-1is a frame of interest to the operator. Accordingly, the video monitoring system may allocate more resources to frame520-1(e.g., bandwidth and/or processing resources) to improve the presentation of the video stream in frame520-1, and reduce resources allocated to other streams corresponding to frames which are not the focus (e.g., in the peripheral vision) of the operator.

FIG. 6is a diagram of functional components of camera110, display130, and VMS150. The functional components of camera110may be implemented, for example, via controller240executing instructions stored in memory245. Alternatively, some or all of the functional components included in camera110may be implemented via hard-wired circuitry. The functional components of display130and/or VMS150may be implemented, for example, via processor320executing instructions stored in memory330. Alternatively, some or all of the functional components included in display130and/or VMS150may be implemented via hard-wired circuitry.

As shown inFIG. 6, camera110may include a sensor array manager610, an image processor620, an encoder630, and a client interface640; monitoring station125may include a decoder650and a display interface660; and VMS150may include an eye tracker interface670, a resource manager680, a camera database (DB)685, and a camera interface690.

A video stream from camera110may follow the following processing path to display130. Sensor array manager610directs sensor array220to capture a set of images for the video stream. Image processor620may perform image processing on the captured images, such as noise reduction operations and/or scaling operations. Encoder630may then compress the images using a codec such as, for example, MPEG-4. Client interface640may then encapsulate the encoded images into a container, such as, for example, MPEG-4 Part 14 (MP4) and may transmit the contained encoded images via data units across network120to monitoring station125for display on display130. Decoder650may retrieve the encoded images from the container, may decode the images, and may provide the decoded images to display interface660. Display interface660may store the decoded images in a buffer and may stream the decoded images from the buffer as a video stream on display130.

Resource manager680may manage resources associated with environment100. For example, resource manager680may manage network resources associated with transmission of data from cameras110to monitoring stations125and associated displays130across network120, and/or processor and memory resources associated with cameras110, monitoring stations125, and/or displays130. Resource manager680may instruct camera110to reduce a bit rate associated with a video stream from camera110to display130for a low interest area determined based on a historical gaze heat map. Eye tracker interface670may be configured to communicate with eye tracker140. For example, eye tracker interface670may obtain information identifying a gaze area associated with a particular video stream from eye tracker140using a particular Application Programming Interface (API) associated with eye tracker140.

Resource manager680may collect gaze information from eye trackers140via eye tracker interface670and may, over a time period, generate a historical gaze heat map based on the collected gaze information. Resource manager580may store the generated historical gaze heat map in camera DB685. Resource manager580may determine one or more low interest areas, and/or one or more high interest areas, for camera110based on the generated historical gaze heat map. Resource manager580may then instruct camera to reduce a bit rate for the one or more low interest areas, and/or to increase a bit rate for one or more high interest areas.

Camera DB685may store information relating to particular cameras110. Exemplary information that may be stored in camera DB685is described below with reference toFIG. 7A. Camera interface690may be configured to communicate with cameras110and may send instructions from resource manager680via a particular API associated with camera110.

Sensor array manager610may store, manage, and/or apply one or more sensor array parameters. For example, sensor array manager610may store parameters governing whether a particular sensor in sensor array220should be turned on or off, a sampling rate for a particular sensor, a sensitivity factor for a particular sensor, and/or another type of sensor parameters. Moreover, sensor array manager610may store one or more setting for sensor array220to determine a type of image captured by sensor array220. For example, a first setting may correspond to a regular image, a second setting may correspond to a wide angle or panoramic image, a third setting may correspond to a low light setting, etc. Sensor array manager610may receive an instruction from VMS150to adjust one or more of the stored parameters in order to adjust a bit rate in a low interest area of sensor array220based on a gaze heat map determined by VMS150.

Image processor620may store, manage, and/or apply one or more image processing parameters. For example, image processor620may store parameters relating to a noise reduction process, such as a low pass filter, parameters relating to a scaling process, and/or other types of image processing parameters that may be used to change a bit rate associated with a region of a video stream. Image processor620may receive an instruction from VMS150to adjust one or more of the stored parameters in order to adjust a bit rate in a low interest area of a video stream based on a gaze heat map determined by VMS150.

Encoder630may store, manage, and/or apply one or more encoding parameters, including intra-frame encoding parameters and inter-frame encoding parameters. For example, encoder630may store a quantization parameter (QP) for particular regions and/or objects of a video stream, store a set of coefficients for a discrete cosine transform (DCT), a Mean Absolute Difference (MAD) of Prediction Error parameter, and/or other encoding parameters. Encoder630may receive an instruction from VMS150to adjust one or more of the stored encoding parameters in order to adjust a bit rate in a low interest area of a video stream based on a gaze heat map determined by VMS150.

Client interface640may store, manage, and/or apply one or more image transmission parameters. For example, client interface640may store a Quality of Service (QoS) parameter. Client interface640may receive an instruction from VMS150to adjust one or more of the stored encoding parameters in order to adjust a bit rate in a low interest area of a video stream based on a gaze heat map determined by VMS150.

AlthoughFIG. 6shows exemplary functional components of camera110, display130, and VMS150, in other implementations, camera110, display130, or VMS150may include fewer functional components, different functional components, differently arranged functional components, or additional functional components than depicted inFIG. 6. Additionally, any one of the components (or any group of components) of camera110, display130, and VMS150may perform functions described as performed by one or more other functional components of camera110, display130, and VMS150.

FIG. 7Ais a diagram of exemplary components of the camera database ofFIG. 6. As shown inFIG. 7A, camera DB685may store one or more camera records701. Each camera record701may store information relating to a particular camera110. Camera record701may include a camera identifier (ID) field710, a gaze heat map720, a low interest area field730, and a bit rate reduction field740.

Camera ID field710may store one or more IDs associated with a particular camera110. For example, camera ID may store a Media Access Control (MAC) address for the particular camera110, an Internet Protocol (IP) address for the particular camera110, a name assigned to the particular camera110by VMS150, and/or another type of ID. Furthermore, camera ID field710may store make and model information for the particular camera110and/or a software version installed on the particular camera110. Moreover, camera ID field710may include authentication information for the particular camera110that enables VMS150to establish a secure connection with the particular camera110.

Gaze heat map720may store gaze heat map information for the particular camera110. For example, each location (e.g., pixel, set of pixels, etc.) of video stream may be associated with a set of data points indicating gaze information. Exemplary information that may be stored in gaze heat map720is described below with reference toFIG. 7B.

Low interest area field730may store information identifying one or more low interest areas for the particular camera110. Furthermore, one or more high interest areas may be identified in low interest area field730as well. Additionally, low interest area field730may identify one or more low interest areas for a particular time of day, a particular day of week, a particular pan, tilt, zoom, rotation, and/or image type setting for the particular camera110, a particular event type, a particular operator, and/or other types of parameters. Each low interest area may be identified as a set of pixels in the video stream from the particular camera110.

Bit rate reduction field740may identify one or more bit rate reduction factors that are to be applied in particular situations. Furthermore, bit rate reduction field740may identify one or more bit rate reduction factors that are currently being applied to a video stream associated with the particular camera110. For example, bit rate reduction field740may identify one or more sensor array manager610parameters, one or more image processor parameters620, one or more encoder parameters, and/or one or more client interface parameters640. The encoder parameters may include different standard encoding profiles which can be adjusted to effect bitrate. For example, when using the H.264 video encoding standard, encoding profiles which may be selected include Baseline, Extended, Main, High, High 10, High 4:2:2, and High 4:4:4 Predictive. Additionally or alternatively, lower level encoding parameters may be adjust to further adjust bitrate. For example, for MPEG encoding standards, the quantization scaling matrices may be selected to increase quantization to reduce the bit rates for encoded intra-frames. Moreover, change threshold levels may be adjusted to change compression rates for encoded inter-frames. For example, the threshold for movement may be raised when encoding P-frames and/or B-frames, and thus less changes are encoded which would lower the bitrate for an encoded video stream.

AlthoughFIG. 7Ashows exemplary components of camera DB685, in other implementations, camera DB685may store fewer components, different components, differently arranged components, or additional components than depicted inFIG. 7A.

FIG. 7Billustrates exemplary components of gaze heat map720. As shown inFIG. 7B, gaze heat map720may include a set of location records750. Each location record750may store gaze information and additional information for a particular location in the video stream of camera110associated with camera record701. Location record750may include a location field752, a value field754, and one or more gaze information records760.

Location field752may identify a particular location. For example, location field752may identify a pixel, a subpixel, a set of pixels, an area, and/or another subunit of a video frame from camera110. Value field754may store one or more gaze heat map values associated with the particular location. For example, value field754may store a normalized value of the number of times gaze point410, or another measure of gaze information (e.g., foveal vision area420, an area with a designated radius around gaze point410, etc.) fell within the particular location with a particular time period. If the particular time period is ongoing, value field754may continue to be updated as new gaze information is received from eye tracker140.

Furthermore, value field754may store additional values. For example, value field754may store values corresponding to a subset of gaze information records760filtered based on a parameters, such as a time of day, a camera setting, etc. Moreover, value field754may store one or more threshold for determining a low interest area, and/or one or more threshold for determining a high interest area. For example, different thresholds may be set based on a desired reduction in bandwidth and/or processor load for the video stream from camera110. If a higher reduction in bandwidth and/or processor load is selected, a higher threshold may be selected, which may result in more locations being filtered out of gaze heat map720, resulting in a higher low interest area.

Each gaze information record760may store information relating to a particular gaze information data point associated with the particular location. Gaze information record760may include a gaze point field762, a timestamp field764, a camera setting field766, an operator field768, and an event type field770.

Gaze point field762may identify a particular gaze point data point. For example, gaze point field762may store gaze information data received from eye tracker140. Timestamp field764may include a timestamp (e.g., time and date) for the particular gaze point data point. Camera setting field766may identify one or more camera settings associated with the particular gaze point data point. The camera settings information may be received from a computer device associated with display130and/or from camera110via camera interface690. The camera settings information may include information identifying a pan setting, a tilt setting, a rotation setting, and/or a zoom setting. Furthermore, the camera settings information may identify a particular image type setting, such as a normal angle setting, a wide angle setting, a fisheye lens setting, a color filter setting, a light source setting, a sensor array setting (e.g., visible light, infrared light, etc.), and/or another type of image type setting.

Operator field768may identify a particular operator associated with the gaze point data point. For example, different operators may prefer to look at different parts of the video frame. The operator may be identified, for example, based on the login information obtained from the computer device associated with display130. Event type field770may identify an event type associated with the particular gaze point data point. For example, VMS150may obtain data that a particular sensor has been activated in connection with camera110at the time the particular gaze point data point was obtained, such a motion sensor, a door sensor, a fire alarm sensor, a microphone, a people counter sensor, a vehicle sensor in a garage, and/or another type of sensor. Furthermore, VMS150may be configured to receive calendar information associated with monitored area106, such as events scheduled for the particular area (e.g., a conference room being booked for a meeting, maintenance being scheduled, a fire alarm being scheduled, etc.).

AlthoughFIG. 7Bshows exemplary components of gaze heat map720, in other implementations, gaze heat map720may store fewer components, different components, differently arranged components, or additional components than depicted inFIG. 7B.

FIG. 8is a flowchart of a process for controlling a bit rate based on a gaze area according to an implementation described herein. In one implementation, the process ofFIG. 8may be performed by VMS150. In other implementations, some or all of the process ofFIG. 8may be performed by another device or a group of devices separate from and/or including VMS150, such as camera110and/or monitoring station125.

The process ofFIG. 8may include receiving a video stream from a camera (block810) and displaying the video stream on a display (block820). For example, an operator may log into a computer device associated with monitoring station125and/or display130and may log into VMS150to configure one or more cameras110. VMS150may configure camera110to provide a video stream of monitored area106to display130and display130may continue to receive video stream data from camera110and display the video stream data.

Gaze information may be obtained for an operator watching the display via an eye tracking sensor (block830). For example, eye tracker140may monitor the operator's eyes to determine gaze point410and/or foveal vision area420and determine the gaze area based on the determined gaze point410and/or foveal vision area420. Information identifying the determined gaze area may be provided to resource manager680of VMS150. Alternatively, raw data from eye tracker140may be provided to VMS150and VMS150may determine the gaze area based on the raw data.

A historical gaze heat map for the video stream may be generated based on the obtained gaze information (block840). For example, VMS150may add up the gaze information for each location of the video stream and may generate a value for each location indicating how many times, or how frequently, the operator's gaze point410(and/or foveal vision area420) fell on the location. VMS150may use the generated values to identify low interest (and/or high interest) areas for the video stream.

In some implementations, VMS150may generate a graphical representation of the gaze heat map that may be available for viewing by the operator, or by an administrator. VMS150may assign a shading or a color to each value in the gaze heat map in the graphical representation. For example, a dark shading or color may be assigned to a high value corresponding to a location that is frequently associated with the operator's gaze point410and a light shading or color may be assigned to a low value corresponding to a location that is associated with the operator's gaze point410less frequently.

A low interest area for the video stream may be determined based on the generated historical gaze heat map (block850). For example, VMS150may analyze the historical gaze heat map to identify areas that have low historical gaze occurrences. A low interest area threshold may be set for a particular location in the video stream, such as a pixel or a set of pixels, to determine whether the location satisfies the requirements for a low interest area. As an example, if an operator looks at the location less than a threshold number of times per unit time (e.g., per hour), the location may be designated as a low interest area. As another example, if the operator looks at the location less than a threshold percentage of time out of the total amount of time watching the video stream, the location may be designated as a low interest area. A similar procedure may be used to identify a high interest area. For example, if an operator looks at the location more than a threshold number of times per unit time (e.g., per hour), and/or if the operator looks at the location more than a threshold percentage of time out of the total amount of time watching the video stream, the location may be designated as a high interest area.

As mentioned above with respect to gaze heat map field720of camera DB685, each location in the gaze heat map may include additional information, such as time and date information, camera setting information, operator information, and/or event type information for each gaze information data point. The additional information may be used to refine the gaze heat map with respect to any parameter included in the gaze heat map. A refined gaze heat map may be used to generate low interest (and/or high interest) areas with respect to particular parameters.

Thus, different low interest area (and/or high interest areas) may be selected for different filtering criteria for the historical gaze heat map. For example, different low interest areas may be selected for different times of day, different days of the week, different camera settings, different operators, and/or different event types.

Furthermore, in some implementations, a graphical representation of a refined gaze heat map may be generated and may available for viewing by the operator or by an administrator. Thus, for example, a graphical representation of the gaze heat map for a particular time of day may be generated.

The camera may be instructed to decrease the bit rate of the video stream in the low interest area (block860). For example, VMS150may select one or more bit rate reduction factors from bit rate reduction field740for the particular camera110, such as a sensor array bit rate reduction factor, an image processing bit rate reduction factor, an encoding bit rate reduction factor, and/or an image transmission bit rate reduction factor. As an example, VMS150may select to adjust a sampling rate of a subset of sensors in sensor array220associated with the low interest area, to down-sample (e.g. lower the resolution) the low interest area, to increase a noise reduction process in the low interest area, to increase an encoding compression parameter in the low interest area, and/or to adjust another parameter that may result in a reduced bit rate in the low interest area.

FIGS. 9A-9Dare diagrams of exemplary gaze heat map scenarios according to one or more implementations described herein. As shown inFIG. 9A, video frame set901on display130may include video streams from four different cameras110. Frame910displays a video stream from a camera monitoring an entrance lobby of an office building. The lobby includes entrance doors, a vestibule, and a hallway. Frame920displays a video stream from a camera monitoring a parking lot with a pathway to a door. Frame930displays a video stream from a camera monitoring an office suite with cubicles. Frame940displays a video stream from a camera monitoring a back door that opens into an alley.

FIG. 9Billustrates a set of gaze heat maps902generated for video frame set901. Set of gaze heat maps902may correspond to a graphical representation of information stored in gaze heat map720. If a value at a particular location752of gaze heat map720is below a threshold, the location may be designated as not being included in the gaze meat map720in a graphical representation or for purposes of determining a low interest area. A graphical representation of gaze heat map720may be generated and displayed to an operator or administrator based on request. Additionally, identified low interest areas may be displayed along with the gaze heat map. In some implementations, a user or an operator may select one or more low interest areas manually, or may adjust identified low interest areas manually, via a graphical user interface that displays the graphical representation of gaze heat map720.

Gaze heat map920corresponds to frame910. Gaze heat map920indicates that the operator spends most of the time watching the main doors and the path that people walk from the main doors down the vestibule and the side hallway. Frame910includes a low interest area922and a low interest area924, which correspond to areas of frame910that the operator spends a relatively low amount of time watching, because few people walk or stand in those areas.

Gaze heat map930corresponds to frame912. Gaze heat map930indicates that the operator spends most of the time watching the walkway to the building entrance and the parking lot. Frame912includes a low interest area932, a low interest area934, and a low interest area936, which correspond to areas of frame912that the operator spends a relatively low amount of time watching, such as the grass next to the walkway.

Gaze heat map940corresponds to frame914. Gaze heat map940indicates that the operator spends most of the time watching the cubicles and the path between the cubicles. Frame914includes a low interest area942, a low interest area944, and a low interest area946, which corresponds to areas of frame914that the operator does not watch very often, such as the wall above the cubicles or the wall that separates the cubicles in the foreground.

Gaze heat map950corresponds to frame916. Gaze heat map950indicates that the operator spends most of the time watching the stairway and nearby area, where people step out to smoke cigarettes, and an alley leading to the back area, where nearby foot traffic sometimes passes. Frame916includes low interest area952, which the operator spends a relatively low amount of time watching, as that area does not see much foot traffic.

FIGS. 9C and 9Dillustrate a set of gaze heat maps903and904for different times of day. As shown inFIG. 9C, gaze heat map903may correspond to a morning rush hour time period when employees are entering the building associated with cameras110providing video streams for frames910,912,914, and916. Frame916for the video stream from the camera monitoring a back door that opens into an alley may receive no attention from the operator during the morning hours, as no people may be frequenting this area during this time. Thus, the whole of frame916may be designated as a low interest area954and the bit rate for the entire video stream for frame916may be reduced during these times.

As shown inFIG. 9D, gaze heat map904may correspond to a night time period when employees have left the building associated with cameras110providing video streams for frames910,912,914, and916. Frames910and914may receive less attention from the operator during this time. For example, the operator may continue to monitor the front doors to the lobby, but may not monitor the vestibule of the lobby as no people move about during the night. Thus, low interest areas922and924may be expanded into low interest area928during the night hours. Similarly, the operator may spend less time observing the cubicles in frame914and low interest areas942,944, and946may be changed to low interest areas948and949during the night hours.

This application incorporates by reference herein the following patent applications filed the same day as this patent application: U.S. patent application Ser. No. 15/395,856, titled “Gaze Controlled Bit Rate,” and filed Dec. 30, 2016; U.S. patent application Ser. No. 15/395,403, titled “Alarm Masking Based on Gaze in Video Management System,” filed Dec. 30, 2016; and U.S. patent application Ser. No. 15/395,790, titled “Block Level Frame Rate Control Based on Gaze,” filed Dec. 30, 2016.

For example, while series of blocks have been described with respect toFIG. 8, the order of the blocks may be modified in other implementations. Further, non-dependent blocks and/or signal flows may be performed in parallel.

Further, certain portions, described above, may be implemented as a component that performs one or more functions. A component, as used herein, may include hardware, such as a processor, an ASIC, or a FPGA, or a combination of hardware and software (e.g., a processor executing software). The word “exemplary” as used herein means “as an example for illustration.”

The term “logic,” as used herein, may refer to a combination of one or more processors configured to execute instructions stored in one or more memory devices, may refer to hardwired circuitry, and/or may refer to a combination thereof. Furthermore, a logic may be included in a single device or may be distributed across multiple, and possibly remote, devices.