Patent Description:
Some embodiments described herein relate generally to method and apparatus for video analytics and management regarding moving cameras. In particular, but not by way of limitation, some embodiments described herein relate to methods and apparatus for protecting privacy in videos recorded with moving cameras such as body-worn cameras, vehicle cameras, or cameras of unmanned aerial vehicles (a. drones), and/or the like.

Moving cameras are often used in public spaces and produce massive amounts of data every day. The videos recorded by the moving cameras are often stored locally, uploaded to a server, or streamed live. It is beneficial to protect the privacy of individuals appearing in the videos when these videos are released publically and/or when explicit consent to the video recording is not provided by the individuals. Privacy protection might also be desirable in cases of nonpublic disclosure of the videos. For example, when a video is used as forensic evidence, it would be beneficial to protect the privacy of persons not involved in an incident captured by the video.

Some known systems require manually marking privacy-related information. This can be time consuming, expensive, and hence infeasible for a large amount of data. It is also challenging to automatically select privacy-related information in videos generated by moving cameras. In addition, some known moving cameras can generate additional data along with the series of image frames associated with the video. Such data can include, for example, audio or certain types of metadata such as Global Positioning System ("GPS") coordinates, time stamps, and/or user information. Such data may implicitly violate the privacy of an individual and therefore should also be considered in a privacy protective system. Accordingly, a need exists for methods and apparatus to automatically or semi-automatically protect privacy in data recorded by moving cameras.

<CIT> relates to an efficient scrambling of regions of interest in an image or video to preserve privacy.

According to the invention, an apparatus includes a memory and a processor operatively coupled to the memory. The processor is configured to receive a set of images associated with a video recorded by a moving or a non-moving camera. The processor is configured to detect a structure of a region of interest from a set of regions of interest in an image from the set of images. The processor is configured to classify the structure into a geometric class from a set of predefined geometric classes using machine learning techniques, based on a machine learning model and a set of parameters stored in the memory, alter the region of interest to generate an altered image when the geometric class is associated with an identity of a person, such that privacy associated with the identity of the person is protected, and at least one of send the altered image to a user interface or store the altered image in a standardized format, wherein the geometric class is associated with a background and not associated with the identity of the person when the structure of the region of interest includes a set of substantially straight lines, wherein the processor is further configured to: define a grid placed over the set of images, the grid including a plurality of grid cells, each grid cell from the plurality of grid cells associated with a different region of interest from the plurality of regions of interest in the image, detect a structure of each region of interest in each grid cell from the plurality of grid cells, and classify each grid cell into the set of predefined geometric classes using the set of parameters stored in the memory.

According to another aspect of the invention, a method includes receiving a set of images associated with a video recorded by a moving or a non-moving camera;detecting a structure of a region of interest from a plurality of regions of interest in an image from the set of images;classifying the structure into a geometric class from a set of predefined geometric classes defined using machine learning techniques, based on a machine learning model and a set of parameters stored in the memory;altering the region of interest to generate an altered image when the geometric class is associated with an identity of a person, such that privacy associated with the identity of the person is protected; and at least one of sending the altered image to a user interface or storing the altered image in a standardized format, wherein the geometric class is associated with a background and not associated with the identity of the person when the structure of the region of interest includes a set of substantially straight lines,the method further comprising:defining a grid placed over the set of images, the grid including a plurality of grid cells, each grid cell from the plurality of grid cells associated with a different region of interest from the plurality of regions of interest in the image;detecting a structure of each region of interest in each grid cell from the plurality of grid cells; and classifying each grid cell into the set of predefined geometric classes using the set of parameters.

According to another aspect of the invention, a non-transitory processor-readable medium stores code representing instructions to be executed by a processor operatively coupled to a memory, the code comprising code to cause the processor to:receive a set of images associated with a video recorded by a moving or a non-moving camera;detect a structure of a region of interest from a plurality of regions of interest in an image from the set of images;classify the structure into a geometric class from a set of predefined geometric classes defined using machine learning techniques, based on a machine learning model and a set of parameters stored in the memory;alter the region of interest to generate an altered image when the geometric class is associated with an identity of a person, such that privacy associated with the identity of the person is protected; and at least one of send the altered image to a user interface or store the altered image in a standardized format,wherein the geometric class is associated with a background and not associated with the identity of the person when the structure of the region of interest includes a set of substantially straight lines, wherein the code further comprises code to cause the processor to:define a grid placed over the set of images, the grid including a plurality of grid cells, each grid cell from the plurality of grid cells associated with a different region of interest from the plurality of regions of interest in the image;detect a structure of each region of interest in each grid cell from the plurality of grid cells; and classify each grid cell into the set of predefined geometric classes using the set of parameters.

As used herein, the term "privacy protection" refers to any method that is able to prevent the unauthorized identification of an individual.

As used herein, the term "moving camera" refers to any non-stationary device that is capable of capturing visual data such as images or videos (e.g., a body-worn camera, a vehicle camera, a pan-tilt-zoom (PTZ) camera, an unmanned aerial vehicle camera, etc.). The term "moving camera" also refers to any stationary device that is capable of capturing visual data and that includes moving components. For example, a PTZ camera can adjust its lens to pan, tilt, or zoom while stationary. In some embodiments, a moving camera can capture and/or sense additional data including, for example, audio data, metadata associated with the video (e.g., GPS, time stamps), and/or the like. In some instances, the moving direction of the moving camera is not provided to the geometry-based privacy-protective data management system, the optical flow-based privacy-protective data management system, and/or the combined privacy-protective data management system. In other instances, the moving direction of the moving camera can provided to the geometry-based privacy-protective data management system (e.g., via metadata provided by the moving camera).

As used herein, the term "non-moving" camera refers to static and/or stationary cameras, which can be relocated.

As used herein, the term "body-worn camera" refers to any moving camera worn by a person.

As used herein, the term "vehicle camera" refers to any moving camera installed in a vehicle either as part of a driver assistant system or as an event data recorder (a. dash-cam or black box).

As used herein the term "privacy-related information" refers to any data that are associated with an identity of an individual or an object. In some instances, such privacy-related information can be protected to avoid revealing the identity of the individual or the object.

As used herein, a module can be, for example, any assembly and/or set of operatively-coupled electrical components associated with performing a specific function, and can include, for example, a memory, a processor, electrical traces, optical connectors, software (executing in hardware) and/or the like.

<FIG> is a block diagram illustrating a geometry-based privacy-protective data management system, according to an embodiment. In some embodiments, a moving or a non-moving camera (not shown) records a video that includes a set of images (or video frames) <NUM>(<NUM>) - <NUM>(n), as well as, in some implementations, audio data and metadata (not shown) associated with the set of images <NUM>(<NUM>) - <NUM>(n). The privacy-protective data management controller <NUM> (or a processor within the privacy-protective data management controller <NUM>) is configured to receive the video from the moving or the non-moving camera. Based on a geometry of each region of interest ("ROI") from a set of regions of interest <NUM>-<NUM> in an image <NUM>(<NUM>) from the set of images, the privacy-protective data management controller <NUM> is configured to protect privacy-related information associated with the image <NUM>(<NUM>) that may reveal an identity of a person (e.g., <NUM>) or an object.

Specifically, the privacy-protective data management controller <NUM> detects a structure (or edges, textures) of a region of interest from the set of regions of interest <NUM>-<NUM> in the image <NUM>(<NUM>). Background structures <NUM>-<NUM> such as buildings, cars, roads, and/or the like differ in their geometry to the structure of humans and animals <NUM>. For example, background structures <NUM>-<NUM> such as buildings, cars, roads, and/or the like often have substantially straight lines, while the structure of humans and animals often does not have substantially straight lines. The privacy-protective data management controller <NUM> classifies the structure of each region of interest <NUM>-<NUM> into a geometric class from a set of predefined geometric classes. When the privacy-protective data management controller <NUM> detects a region of interest (ROI) including a set of substantially straight lines (such as ROI <NUM>), the privacy-protective data management controller <NUM> classifies ROI <NUM> as a background. When the privacy-protective data management controller <NUM> detects a ROI including a set of substantially non-structured lines (such as ROI <NUM>), the privacy-protective data management controller <NUM> can classify ROI <NUM> as a foreground (or a target, or a human).

When the geometric class of an ROI is associated with an identity of a person, the privacy-protective data management controller <NUM> then alters (or blurs, scrambles, pixelates, blanks) the ROI (or only a portion of the ROI, such as the face of the person) to remove any privacy-related information in the ROI and generate an altered image, such that privacy associated with the identity of the person is protected. The privacy-protective data management controller <NUM> sends the altered image to a user interface <NUM> or stores the altered image in a standardized format (e.g., in a memory of the privacy-protective data management controller <NUM>, in a memory of a server accessible by the privacy-protective data management controller <NUM>, and/or within any other suitable memory). For example, the privacy-protective data management controller <NUM> can convert the image (or video associated with the image) into a standardized format such as, for example, Audio Video Interleave ("AVI"), Advanced Systems Format ("ASF"), MOV, MPEG-<NUM>, MPEG-<NUM>, and/or the like. The privacy-protective data management controller <NUM> can then store the image (or video associated with the image) in the standardized format.

According to the invention, when the set of images <NUM>(<NUM>)-<NUM>(n) is recorded by the moving or the non-moving camera in a known environment (e.g., city, desert, a ski resort, etc.), the privacy-protective data management controller <NUM> can retrieve, from a memory operatively coupled with the privacy-protective data management controller <NUM>, a set of pre-defined geometric classes associated with the known environment to facilitate with classifying the structure into a geometric class.

The set of predefined geometric classes (e.g., humans, animals, buildings, and/or the like) is defined using machine learning techniques. The privacy-protective data management controller <NUM> can receive manual selections and/or annotations of geometric features from a set of sample images and classify the geometric features into at least one geometric class from the set of predefined geometric classes. The privacy-protective data management controller <NUM> can define a machine learning model and train the machine learning model using a set of parameters (e.g., known annotated data and/or weights associated with a function specified by the machine learning model) and machine learning techniques. Based on the machine learning model and the set of parameters, the privacy-protective data management controller <NUM> automatically classifies the structure in a ROI into a geometric class. In some instances, the machine learning techniques can include, for example, neural networks, deep neural networks, decision trees, random forests, and/or any other suitable machine learning technique.

The privacy-protective data management controller <NUM> defines a grid (not shown) overlaid on and/or placed over each image from the set of images <NUM>(<NUM>)-<NUM>(n). The grid includes a set of grid cells (e.g., ROIs <NUM>-<NUM> are examples of grid cells) and each grid cell from the set of grid cells can define and/or is associated with a different ROI from the set of ROIs in the image. The grid can be in a regular shape or an irregular shape. The grid can be orthogonal or perspective. The size of each grid cell can be substantially the same or different, or arbitrarily chosen to suit a given environment. The privacy-protective data management controller <NUM> detects a structure of each ROI in each grid cell from the set of grid cells. The privacy-protective data management controller <NUM> classifies each grid cell into the set of predefined geometric classes using a set of parameters stored in the memory.

<FIG> is a block diagram illustrating an optical flow-based privacy-protective data management system, according to an example not covered by the claims. In some non-claimed examples, a camera (not shown), moving in a direction <NUM>, records a video that includes a set of images <NUM>(<NUM>) - <NUM>(n). In some implementations, the moving camera also records audio data and/or metadata (not shown). The privacy-protective data management controller <NUM> (or a processor within the privacy-protective data management controller <NUM>) can be configured to receive the video from the moving camera. As the camera moves in direction <NUM>, a first set of feature points <NUM> may appear to move in the recorded video in direction <NUM>, and a second set of feature points <NUM> may appear to move in the recorded video in direction <NUM> different from direction <NUM>. Based on the motion of the first set of feature points <NUM> relative to the motion of the second set of feature points <NUM>, the privacy-protective data management controller <NUM> can be configured to protect privacy-related information associated with the image <NUM>(<NUM>) that may reveal an identity of a person (e.g., <NUM>) or another object of interest (e.g., a vehicle).

For example, in a situation when a camera is worn on a policeman (not shown) who is chasing a suspect <NUM>, the camera, moving in a direction <NUM> based on the policeman's movement, records a video that includes a set of images <NUM>(<NUM>) - <NUM>(n). The privacy-protective data management controller <NUM> can be configured to receive the set of images <NUM>(<NUM>) - <NUM>(n) from the moving camera. The privacy-protective data management controller <NUM> (or a processor therein) can select a set of feature points <NUM>-<NUM> in an image <NUM>(<NUM>) from the set of images <NUM>(<NUM>) - <NUM>(n). In some implementations, the privacy-protective data management controller <NUM> can select the set of feature points based on a texture (e.g., pattern, appearance, design, and/or the like) associated with each region of interest from the set of regions of interest in the image <NUM>(<NUM>). In some implementations, the privacy-protective data management controller <NUM> can select the set of feature points randomly within each ROI from the set of ROIs in the image. In some implementations, the privacy-protective data management controller <NUM> can select the center of each ROI from the set of ROIs in the image as the set of feature points. In some implementations, the privacy-protective data management controller <NUM> can apply the edge, corner, blob, or ridge detectors to select the set of feature points.

The privacy-protective data management controller <NUM> can then define a moving direction of each feature point from the set of feature points <NUM>-<NUM>. In some situations, a first subset of feature points <NUM> from the set of feature points <NUM>-139moves in a first direction, and a second subset of feature points <NUM> from the set of feature points <NUM>-139moves in a second direction different from the first direction. In this example, because the camera moves in the second direction <NUM>, the buildings <NUM>-<NUM> in the video appear to move in the first direction <NUM> opposite to the second direction <NUM>. The second set of feature points <NUM> appears to move in the same direction <NUM> as the camera because the policeman (not shown) is moving in the same direction as the person <NUM>. In another example, the second set of feature points <NUM> may appear to move in a different direction (but not at the same speed) as the camera or may appear still in the set of images <NUM>(<NUM>) - <NUM>(n) if the camera and the person <NUM> are moving at substantially the same speed or if the camera is following the person <NUM>.

In some not claimed examples, the privacy-protective data management controller <NUM> can determine a number of feature points from the first subset of feature points <NUM> that appear to move in the first direction <NUM> and a number of feature points from the second subset of feature points <NUM> that appear to move in the second direction <NUM>. A set of feature points that is associated with the background of an image (i.e., moves in an opposite direction from the camera) often has a greater number of feature points than a set of feature points that is associated with the foreground of the image (i.e., moves in the same direction as the camera or in a different direction as the first subset of feature points <NUM>). In other words, when the number of feature points of the first subset of feature points is greater than the number of feature points of the second subset of feature points, the privacy-protective data management controller <NUM> can determine and/or infer that the first set of feature points is associated with the background and the second set of feature points is associated with the person <NUM> (or other foreground object of interest). The privacy-protective data management controller <NUM> can alter the ROIs associated with each feature point from the second subset of feature points <NUM> to produce an altered image to protect privacy-related information included in the region of interest associated with each feature point from the second subset of feature points <NUM>. The privacy-protective data management controller <NUM> can send the altered image to a user interface <NUM> or store the altered image in a standardized format.

According to the invention the ROIs are defined based on a grid (as described above). Specifically, the privacy-protective data management controller <NUM> defines a grid (not shown) overlaid and/or placed over each image from the set of images <NUM>(<NUM>)-<NUM>(n). The grid includes a set of grid cells (e.g., ROIs <NUM>-<NUM> are examples of grid cells). A feature point can then be defined and/or identified for each ROI by the privacy-protective data management controller <NUM> (or a processor therein). For example, in some instances, the feature point can be a center point in each ROI. In other instances, the feature point can be randomly selected in each ROI. In still other instances, the feature point can be selected for each ROI in any other suitable manner. The grid can be in a regular shape or an irregular shape. The grid can be orthogonal or perspective. The size of each grid cell can be substantially the same or different, or arbitrarily chosen to suit a given environment. The privacy-protective data management controller <NUM> can define a trajectory associated with each feature point defined in each grid cell from the set of grid cells and determine a set of characteristics associated with the trajectory of each feature point. The set of characteristics can include a length of a trajectory, a moving direction of the trajectory, and/or the like. When the set of characteristics associated with a second set of feature points (e.g., <NUM>) is less coherent and/or uniform than the set of characteristics associated with a first set of feature points (e.g., <NUM>), the ROIs associated with the first set of feature points (e.g., <NUM>) are likely and/or can be inferred to be associated with a substantially stationary background (since such movement is solely based on the movement of the camera) and the ROIs associated with the second set of feature points (e.g., <NUM>) are more likely to be associated with a foreground object (e.g., such as a person or vehicle).

For example, in some situations, the closer an object (or a person) is to the camera, the longer the trajectory. In other words, when the length of the trajectory of the second set of feature points (e.g., <NUM>) is greater than the length of the trajectory of the first set of feature points (e.g., <NUM>), the privacy-protective data management controller <NUM> can determine and/or infer that the second set of feature points (e.g., <NUM>) is closer to the camera than the first set of feature points (e.g., <NUM>). Thus, the second set of feature points (e.g., <NUM>) is more likely to be associated with a foreground object (e.g., a person or vehicle). The privacy-protective data management controller <NUM> can alter the ROIs associated with the second set of feature points (e.g., <NUM>) to produce the altered image to protect privacy-related information included in the ROIs associated with the second set of feature points (e.g., <NUM>).

<FIG> is a block diagram illustrating a combined privacy-protective data management system, according to an embodiment. In some embodiments, the privacy-protective data management controller <NUM> can combine at least two of the optical flow-based non-claimed example illustrated with regards to <FIG>, the geometry-based embodiment illustrated with regards to <FIG>, and an object detector example not covered by the claims (described in further detail herein), to improve the reliability of the privacy protection in different use cases.

For example, a video (or a set of images <NUM>(<NUM>)-<NUM>(n)), recorded by a moving camera (not shown), can include non-moving and structured buildings <NUM>-<NUM>, a set of structured cars <NUM>-<NUM> moving in different directions, and a person of interest <NUM> moving in a direction <NUM>. The moving camera can be, for example, on a moving drone configured to record activities of the person of interest <NUM>. The privacy-protective data management controller <NUM> (or processor therein) can first determine that a first set of feature points of a first set of ROIs <NUM>-<NUM> appear to be moving in a direction <NUM> opposite to the direction <NUM> of the camera, while a second set of feature points of a second set of ROIs <NUM> appear to be moving in same direction <NUM> of the camera. In addition, the first set of feature points <NUM> include a greater number of feature points than the second set of feature points <NUM>, indicating that the second set of feature points <NUM> is more likely to be associated with an object of interest (e.g., a person or vehicle). The privacy-protective data management controller <NUM> can calculate a confidence score of privacy protection and determine if the confidence score meets a predefined criterion. If the confidence score meets the predefined criterion, the privacy-protective data management controller <NUM> can alter ROIs associated with the second set of feature points <NUM> to produce an altered set of images such that the privacy related information associated with the second feature points <NUM> is protected. In some implementations, the confidence score of privacy protection can be a numerical value indicating a likelihood of all or a portion of privacy-related information in the video is protected. For example, a confidence score of <NUM>% indicates that the likelihood of having no privacy-related information remaining in the video is <NUM>%. The confidence score can be calculated based on a set of sample images and manually defined confidence score of each sample image of the set of sample images.

In some not claimed examples, if the confidence score does not meet the predefined criterion, the privacy-protective data management controller <NUM> can use the geometry-based embodiment illustrated with regards to <FIG>, and/or an object detector example not covered by the claims to improve the reliability of the privacy protection. In other implementations, the privacy-protective data management controller <NUM> can determine to use the geometry-based embodiment illustrated with regards to <FIG>, and/or an object detector example not covered by the claims to improve the reliability of the privacy protection, regardless of the confidence score of the optical flow-based non-claimed example illustrated above. For example, the privacy-protective data management controller <NUM> can detect that the structure associated with ROIs <NUM>-<NUM> includes a set of substantially straight lines, and the structure associated with ROI <NUM> does not include a set of substantially straight lines. Based on the structure, the privacy-protective data management controller <NUM> can classify ROIs <NUM>-<NUM> as background (or not an object of interest) and ROI <NUM> as a foreground (or an object of interest). The privacy-protective data management controller <NUM> can classify the ROIs <NUM>-<NUM> as background even though the feature points of ROIs <NUM>-<NUM> may indicate that they are not background. Thus, by using both the optical flow and geometry-based methods, the privacy-protective data management controller <NUM> can better and more definitively identify specific objects of interest (e.g., distinguish moving people from moving vehicles and stationary buildings). The privacy-protective data management controller <NUM> can again calculate the confidence score of privacy protection and determine if the confidence score meets a predefined criterion. If the confidence score meets the predefined criterion, the privacy-protective data management controller <NUM> can alter ROI <NUM> to produce an altered set of images such that the privacy related information associated with ROI <NUM> is protected.

In some not claimed examples, if the confidence score does not meet the predefined criterion, the privacy-protective data management controller <NUM> can use an object detector method and/or process to improve the reliability of the privacy protection. In other implementations, the privacy-protective data management controller <NUM> can determine to use the object detector method and/or process regardless of the confidence score determined. The privacy-protective data management controller <NUM> can detect a person, a face, or an object using machine learning techniques. For example, the privacy-protective data management controller <NUM> can define a machine learning model and train the machine learning model using a set of parameters and/or data that include manually marked and/or annotated ROIs in a set of sample images of, for example, a person, a face, and/or an object. Such parameters can be used to optimize the machine learning model. Based on the machine learning model and the set of parameters, the privacy-protective data management controller <NUM> can automatically detect certain features in an image as a person, a face, or an object. Once detected, the privacy-protective data management controller <NUM> can track such features in the set of images of the video and alter these feature to produce an altered set of images. The privacy-protective data management controller <NUM> can again calculate the confidence score of privacy protection and determine if the confidence score meets a predefined criterion. If the confidence score privacy protection meets the predefined criteria, the privacy-protective data management controller <NUM> can alter the ROIs associated with an object of interest (e.g., a person or vehicle) to produce an altered image(s). The privacy-protective data management controller <NUM> can send the altered set of images to the user interface <NUM> or store the altered set of images in a standardized format.

In some not claimed examples, the privacy-protective data management controller <NUM> can detect if a ROI is associated with an object of interest based at least in part on a physical motion limit of the object of interest. For example, if a feature in the ROI moves quickly to a height of <NUM> feet (i.e., as movement is monitored between the set of images), the privacy-protective data management controller <NUM> can determine that the feature is in not associated with a person or car because a person or car is not able to jump to a height of <NUM> feet.

In some not claimed examples, the privacy-protective data management controller <NUM> can determine if a smaller area of ROI within a previously-detected and larger area of ROI is associated an identity of a person. For example, the privacy-protective data management controller <NUM> can determine that ROI <NUM> is associated with an identity of a person. The privacy-protective data management controller <NUM> can further detect that only a smaller area of ROI <NUM> within the ROI <NUM> is associated with the identity of the person. Thus, the privacy-protective data management controller <NUM> can alter the smaller area of ROI <NUM> to protect the privacy without altering other area of ROI <NUM> where privacy information is not disclosed.

<FIG> is a block diagram illustrating a privacy-protective data management controller <NUM>. The privacy-protective data management controller <NUM> can be structurally and/or functionally similar to privacy-protective data management controller <NUM> in <FIG>. In some embodiments, controller <NUM> is a hardware device (e.g., compute device, server, smart phone, tablet, etc.). Controller <NUM> can include a processor <NUM>, a memory <NUM>, a communications interface <NUM>, a geometry detector <NUM>, an optical flow analyzer <NUM>, an object detector <NUM>, and a scrambler <NUM>.

Each module or component in controller <NUM> can be operatively coupled to each remaining module or component. Each of the geometry detector <NUM>, the optical flow analyzer <NUM>, the object detector <NUM>, and the scrambler <NUM> in controller <NUM> can be any combination of hardware and/or software (stored and/or executing in hardware such as processor <NUM>) capable of performing one or more specific functions associated with that module. In some implementations, a module or a component in controller <NUM> can include, for example, a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), software (stored and/or executing in hardware such as processor <NUM>) and/or the like.

The processor <NUM> can be any suitable hardware processing device such as, for example, a general purpose processor, a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), a combination thereof, and/or the like. In some instances, the processor <NUM> can include and/or execute software-based modules (e.g., a module of computer code executed at processor <NUM>, a set of processor-readable instructions executed at processor <NUM>), and/or a combination of hardware- and software-based modules. The processor <NUM> can be or can include any processing device or component configured to perform the data collecting, processing and transmitting functions as described herein. The processor <NUM> can be configured to, for example, write data into and read data from the memory <NUM>, and execute the instructions stored within the memory <NUM>. Processor <NUM> can also be configured to execute and/or control, for example, the operations of the communications interface <NUM>, the geometry detector <NUM>, the optical flow analyzer <NUM>, the object detector <NUM>, and the scrambler <NUM>. In some implementations, based on the methods or processes stored within the memory <NUM>, the processor <NUM> can be configured to execute the geometry-based privacy-protective data management method, the optical flow-based privacy-protective data management method, and/or the combined privacy-protective data management method, as described in <FIG> respectively.

The memory <NUM> can be, for example, a random-access memory (RAM) (e.g., a dynamic RAM, a static RAM), a flash memory, a removable memory, a read only memory (ROM), and/or so forth. In some embodiments, the memory <NUM> can include, for example, a database, process, application, virtual machine, and/or some other software modules (stored and/or executing in hardware) or hardware modules configured to execute the geometry-based privacy-protective data management method, the optical flow-based privacy-protective data management method, and the combined privacy-protective data management method. In such implementations, instructions for executing the geometry-based privacy-protective data management method, the optical flow-based privacy-protective data management method, and the combined privacy-protective data management method and/or the associated methods can be stored within the memory <NUM> and executed at the processor <NUM>.

The communications interface <NUM> can include and/or be configured to manage one or multiple ports (not shown in <FIG>) of the privacy-protective data management controller <NUM>. In some instances, for example, the communications interface <NUM> (e.g., a Network Interface Card (NIC) and/or the like) can be operatively coupled to devices (e.g., user input devices not shown in <FIG>) and can actively communicate with a coupled device or over a network (e.g., communicate with end-user devices, host devices, servers, etc.). The communication can be over a network such as, for example, a Wi-Fi or wireless local area network ("WLAN") connection, a wireless wide area network ("WWAN") connection, a cellular connection and/or the like. A network connection can be a wired connection such as, for example, an Ethernet connection, a digital subscription line ("DSL") connection, a broadband coaxial connection, a fiber-optic connection and/or the like. In some embodiments, the communications interface <NUM> can be configured to, among other functions, receive data and/or information, and send commands, and/or instructions.

The geometry detector <NUM> is configured to protect, based on a geometry of each region of interest ("ROI") from a set of regions of interest in an image from a set of images, privacy-related information associated with an object of interest in the image (e.g., that may reveal an identity of a person or an object). In some implementations, a moving or a non-moving camera (not shown) records a video that includes a set of images, as well as, in some implementations, audio data and/or metadata (not shown) associated with the set of images. The geometry detector <NUM> is configured to receive the video from the moving or the non-moving camera via the communications interface <NUM>. The geometry detector <NUM> detects a structure (or edges, textures) of a region of interest from the set of regions of interest in the image (e.g., as described with respect to <FIG>). Background structures such as buildings, cars, roads, and/or the like differ in their geometry to the structure of humans and animals. For example, background structures such as buildings, cars, roads, and/or the like often have substantially straight lines, while the structure of humans and animals often does not have substantially straight lines. The geometry detector <NUM> classifies the structure of each region of interest into a geometric class from a set of predefined geometric classes. For example, when the geometry detector <NUM> detects a ROI comprising a set of substantially straight lines (such as ROI <NUM> of <FIG>), the geometry detector <NUM> classifies ROI as a background. When the geometry detector <NUM> detects a ROI comprising a set of substantially non-structured lines (such as ROI <NUM> of <FIG>), the geometry detector <NUM> can classify ROI <NUM> as a foreground (or a target, or a human).

When the geometric class of an ROI is associated with an object of interest (e.g., an identity of a person), the geometry detector <NUM> then alters (or blurs, scrambles, pixelates, blanks) the ROI (or only the face of the person) to remove any privacy-related information in the ROI and generate an altered image, such that privacy associated with the identity of the person is protected. The geometry detector <NUM> sends the altered image to a user interface or stores the altered image in a standardized format.

In some implementations, when the set of images are recorded by the moving or the non-moving camera in a known environment (e.g., a city, desert, ski resort or other known environment), the geometry detector <NUM> can retrieve, from a memory operatively coupled with the geometry detector <NUM>, a set of pre-defined geometric classes associated with the known environment to facilitate with classifying the structure into a geometric class. For example, the set of pre-defined geometric classes can be defined based on a set of sample images recorded by a non-moving camera or a moving camera in a known or an unknown environment. The environment in which the set of images are recorded can be the same as or different from the environment in which the set of sample images are recorded.

The set of predefined geometric classes (e.g., humans, animals, buildings, and/or the like) is defined using machine learning techniques. The geometry detector <NUM> can receive manual selections of geometric features from a set of sample images and classify the geometric features into at least one geometric class from the set of predefined geometric classes. In some implementations, the set of sample images (or parameters to train the machine learning model) can be recorded by a non-moving camera in a known or an unknown environment. In some implementations, the set of sample images can be recorded by a moving camera in a known or an unknown environment. The geometry detector <NUM> can define a machine learning model and train the machine learning model using a set of parameters (as defined by the set of sample images) and machine learning techniques (as described above) with the set of sample images. Based on the machine learning model and the set of parameters, the geometry detector <NUM> automatically classifies the structure in a ROI into a geometric class.

According to the invention, the geometry detector <NUM> defines a grid (not shown) placed over and/or overlaid on each image from the set of images The grid includes a set of grid cells (not shown) and each grid cell from the set of grid cells is associated with a different ROI from the set of ROIs in the image. The grid can be in a regular shape or an irregular shape. The grid can be orthogonal or perspective. The size of each grid cell can be substantially the same or different, or arbitrarily chosen to suit a given environment. The geometry detector <NUM> detects a structure of each ROI in each grid cell from the set of grid cells. The geometry detector <NUM> classifies each grid cell into the set of predefined geometric classes using a set of parameters stored in the memory.

In some embodiments, a camera (not shown), moving in a direction, records a video that includes a set of images. In some implementations, the moving camera also records audio data and/or metadata (not shown). The optical flow analyzer <NUM> can be configured to receive the video from the moving camera. As the camera moves in a first direction, a first set of feature points may move in the first direction and a second set of feature points may move in a second direction different from the first direction (or in multiple different directions different from the first direction). Based on the motion of the first set of feature points relative to the motion of the second set of feature points, the optical flow analyzer <NUM> can be configured to protect privacy-related information associated with an object of interest (e.g., that may reveal an identity of a person or an object).

The optical flow analyzer <NUM> can be configured to receive the set of images from the moving camera. The optical flow analyzer <NUM> can select a set of feature points in an image from the set of images. In some implementations, the optical flow analyzer <NUM> can select the set of feature points based on a texture (e.g., pattern, appearance, design, and/or the like) associated with each region of interest from the set of regions of interest in the image. In some implementations, the optical flow analyzer <NUM> can select the set of feature points randomly within each ROI from the set of ROIs in the image. In some implementations, the optical flow analyzer <NUM> can select the center of each ROI from the set of ROIs in the image as the set of feature points. In some implementations, the optical flow analyzer <NUM> can apply the edge, corner, blob, or ridge detectors to select the set of feature points.

The optical flow analyzer <NUM> can then define a moving direction of each feature point of the set of feature points. In some situations, a first subset of feature points from the set of feature points moves in a first direction, and a second subset of feature points from the set of feature points moves in a second direction different from the first direction. In this example, because the camera moves in the second direction, stationary objects in the video (e.g., buildings) appear to move uniformly in the first direction opposite to the second direction. Moving objects (e.g., a person or suspect) move in various non-uniform directions as the camera.

In some not claimed examples, the optical flow analyzer <NUM> can determine a number of feature points of the first set of feature points that moves in the first direction and a number of feature points of the second set of feature points that move in the non-uniform directions. A set of feature points that is associated with the background of an image (i.e., moves uniformly in an opposite direction from the camera) often has a greater number of feature points than a set of feature points that is associated with the foreground of the image (i.e., moves in various non-uniform directions and lengths). In other words, when the number of feature points of the first set of feature points is greater than the number of feature points of the second set of feature points, the optical flow analyzer <NUM> can determine that the first set of feature points is associated with the background and the second set of feature points is associated with one or more foreground objects. Additionally, because the movement of the first set of feature points is substantially uniform and the movement of the second set of feature points is substantially non-uniform, the optical flow analyzer <NUM> can identify the first set of feature points as background and the second set of feature points as foreground objects. The optical flow analyzer <NUM> can alter the ROIs associated with each feature point from the second set of feature points to produce an altered image to protect privacy-related information included in the region of interest associated with each feature point from the second set of feature points. The optical flow analyzer <NUM> can send the altered image to the communications interface 290or store the altered image in a standardized format.

According to the invention, the optical flow analyzer <NUM> defines a grid (not shown) placed over each image from the set of images. The grid includes a set of grid cells (not shown) and the optical flow analyzer <NUM> can select a center of each grid cell from the set of grid cells to define the set of feature points. The grid can be in a regular shape or an irregular shape. The grid can be orthogonal or perspective. The size of each grid cell can be substantially the same or different, or arbitrarily chosen to suit a given environment. The optical flow analyzer <NUM> can define a trajectory associated with each grid cell from the set of grid cells and determine a set of characteristics associated with the trajectory of each grid cell. The set of characteristics can include a length of a trajectory, a moving direction of the trajectory, and/or the like. When the set of characteristics associated with a first set of grid cells is less coherent than the set of characteristics associated with a second set of grid cells, the ROIs associated with the first set of grid cells are more likely to be associated with an identity of a person.

For example, in some situations, the closer an object (or a person) is to the camera, the longer the trajectory. In other words, when the length of the trajectory of the second set of feature points (e.g., <NUM> in <FIG>) is greater than the length of the trajectory of the first set of feature points (e.g., <NUM> in <FIG>), the optical flow analyzer <NUM> can determine and/or infer that the second set of feature points is closer to the camera than the first set of feature points. Thus, the second set of feature points is more likely to be associated with a foreground object (e.g., a person or vehicle). The optical flow analyzer <NUM> can alter the ROIs associated with the second set of feature points to produce the altered image to protect privacy-related information included in the ROIs associated with the second set of feature points. In some embodiments, the object detector <NUM> can detect a person, a face, or an object using machine learning techniques. For example, the object detector <NUM> can define a machine learning model and train the machine learning model using a set of parameters and/or data that include manually marked and/or annotated ROIs in a set of sample images of, for example, a person, a face, and/or an object. Such parameters can be used to optimize the machine learning model. Based on the machine learning model and the set of parameters, the object detector <NUM> can automatically detect certain features in an image as a person, a face, or an object. Once detected, the object detector <NUM> can track such features in the set of images of the video and alter these feature to produce an altered set of images. The object detector <NUM> can calculate the confidence score of privacy protection and determine if the confidence score meets a predefined criterion.

In some not claimed examples, the object detector <NUM> can detect if a ROI is associated with an object of interest based at least in part on a physical motion limit of the object of interest. For example, if a feature in the ROI moves quickly to a height of <NUM> feet (i.e., as movement is monitored between the set of images), the object detector <NUM> can determine that the feature is in not associated with a person or car because a person or car is not able to jump to a height of <NUM> feet.

In some not claimed examples, two or more of the geometry detector <NUM>, the optical flow analyzer <NUM>, and/or the object detector <NUM> can detect privacy-related information in the same video file to improve the reliability of the privacy protection in different use cases. For example, each of the geometry detector <NUM>, the optical flow analyzer <NUM>, and the object detector <NUM> can calculate a confidence score of privacy protection and determine if the confidence score meets a predefined criterion. If the confidence score meets the predefined criterion, the scrambler <NUM> can alter the ROIs to produce an altered set of images such that the privacy-related information associated with the ROIs is protected. In some implementations, the confidence score of privacy protection can be a numerical value indicating a likelihood of all or a portion of privacy-related information in the video is protected. For example, a confidence score of <NUM>% indicates that the likelihood of having no privacy-related information remaining in the video is <NUM>%. The confidence score can be calculated based on a set of sample images and manually defined confidence score of each sample image of the set of sample images.

In some not claimed examples, the scrambler <NUM> can alter (or blur, scramble, pixelate, blank) the ROI (or only a portion of the ROI, such as the face of the person) that each of the geometry detector <NUM>, the optical flow analyzer <NUM>, and the object detector detected to remove any privacy-related information in the ROI. The scrambler <NUM> can generate an altered image and send the altered image to the communications interface <NUM>, such that privacy associated with the identity of the person is protected.

<FIG> is a flow chart illustrating a geometry-based privacy-protective data management method <NUM>, according to an embodiment. The geometry-based privacy-protective data management method <NUM> can be executed at, for example, a privacy-protective data management controller such as the privacy-protective data management controller <NUM> shown and described with respect to <FIG> or the privacy-protective data management controller <NUM> shown and described with respect to <FIG>. More specifically, the geometry-based privacy-protective data management method <NUM> can be executed by a processor, such as processor <NUM> of controller <NUM> shown and described with respect to <FIG>.

A moving or a non-moving camera (not shown) records a video that includes a set of images as well as, in some implementations, audio data and metadata (not shown) associated with the set of images. At <NUM>, method <NUM> includes receiving a set of images associated with a video recorded by the moving or the non-moving camera. Based on a geometry of each region of interest ("ROI") from a set of regions of interest in an image from the set of images, method <NUM> protects privacy-related information associated with the image that may reveal an identity of a person or an object.

At <NUM>, the privacy-protective data management controller (such as the privacy-protective data management controller <NUM> in <FIG>) detects a structure (or edges, textures) of a region of interest from the set of regions of interest in the image. Background structures such as buildings, cars, roads, and/or the like differ in their geometry to the structure of humans and animals. For example, background structures such as buildings, cars, roads, and/or the like often have substantially straight lines, while the structure of humans and animals often does not have substantially straight lines. At <NUM>, the privacy-protective data management controller classifies the structure of each region of interest into a geometric class from a set of predefined geometric classes. For example, when the privacy-protective data management controller detects a ROI comprising a set of substantially straight lines the privacy-protective data management controller classifies the ROI as a background. When the privacy-protective data management controller detects a ROI comprising a set of substantially non-structured lines), controller can classify the ROI as a foreground (or a target, or a human).

At <NUM>, when the geometric class of an ROI is associated with an identity of a person (or an object), the privacy-protective data management controller alters (or blurs, scrambles, pixelates, blanks) the ROI (or only the face of the person via face detection algorithms), at <NUM>, to remove any privacy-related information in the ROI and generate an altered image, such that privacy associated with the identity of the person is protected. At <NUM>, the privacy-protective data management controller sends the altered image to a user interface or stores the altered image in a standardized format.

In some implementations, when the set of images are recorded by the moving or the non-moving camera in a known environment (e.g., cities, or desert, or a ski resort), the privacy-protective data management controller can retrieve, from a memory operatively coupled with the privacy-protective data management controller, a set of pre-defined geometric classes associated with the known environment to facilitate with classifying the structure into a geometric class. The set of predefined geometric classes (e.g., humans, animals, buildings, and/or the like) is defined using machine learning techniques. The privacy-protective data management controller can receive manual selections of geometric features from a set of sample images and classify the geometric features into at least one geometric class from the set of predefined geometric classes. The privacy-protective data management controller can define a machine learning model and train the machine learning model using a set of parameters and machine learning techniques. Based on the machine learning model and the set of parameters, the privacy-protective data management controller automatically classifies the structure in a ROI into a geometric class.

The privacy-protective data management controller defines a grid (not shown) overlaid on and/or placed over each image from. The grid includes a set of grid cells (e.g., ROIs <NUM>-<NUM> are examples of grid cells) and each grid cell from the set of grid cells can define and/or is associated with a different ROI from the set of ROIs in the image. The grid can be in a regular shape or an irregular shape. The grid can be orthogonal or perspective. The size of each grid cell can be substantially the same or different, or arbitrarily chosen to suit a given environment. The privacy-protective data management controller <NUM> detects a structure of each ROI in each grid cell from the set of grid cells. The privacy-protective data management controller <NUM> classifies each grid cell into the set of predefined geometric classes using a set of parameters stored in the memory.

<FIG> is s flow chart illustrating an optical flow-based privacy-protective data management method <NUM>, according to an example not covered by the claims. The optical flow-based privacy-protective data management method <NUM> can be executed at, for example, a privacy-protective data management controller such as the privacy-protective data management controller <NUM> shown and described with respect to <FIG> or the privacy-protective data management controller <NUM> shown and described with respect to <FIG>. More specifically, the optical flow-based privacy-protective data management method <NUM> can be executed by a processor, such as processor <NUM> of controller <NUM> shown and described with respect to <FIG>.

In some non-claimed examples, a camera, moving in a direction, records a video that includes a set of images. In some implementations, the moving camera also record audio data and metadata. At <NUM>, the privacy-protective data management controller receives the video and/or a set of images associated with a video recorded by the moving camera. As the camera moves in a first direction, a first set of feature points may appear to move in the first direction and a second set of feature points may appear to move in a second direction different from the first direction. Based on the motion of the first set of feature points relative to the motion of the second set of feature points, the privacy-protective data management controller <NUM> can be configured to protect privacy-related information associated with the image that may reveal an identity of a person or an object.

For example, in a situation when a camera is worn on a policeman (not shown) who is chasing a suspect, the camera, moving in a direction, records a video that includes a set of images. The privacy-protective data management controller receives the set of images from the moving camera. At <NUM>, the privacy-protective data management controller selects a set of feature points in an image from the set of images. In some implementations, the privacy-protective data management controller selects the set of feature points based on a texture (e.g., pattern, appearance, design, and/or the like) associated with each region of interest from the set of regions of interest in the image. In some implementations, the privacy-protective data management controller selects the set of feature points randomly within each ROI from the set of ROIs in the image. In some implementations, the privacy-protective data management controller selects the center of each ROI from the set of ROIs in the image as the set of feature points. In some implementations, the privacy-protective data management controller can apply the edge, corner, blob, or ridge detectors to select the set of feature points.

At <NUM>, the privacy-protective data management controller then defines a first moving direction of each feature point of the first set of feature points, and at <NUM>, the privacy-protective data management controller defines a second moving direction of each feature point from the second set of feature points. In some situations, a first subset of feature points from the set of feature points moves in a first direction, and a second subset of feature points from the set of feature points moves in a second direction different from the first direction. In this example, because the camera moves in the second direction, stationary objects in the video (e.g., buildings) appear to move uniformly in the first direction opposite to the second direction. Moving objects (e.g., a person or suspect) move in various non-uniform directions as the camera.

In some not claimed examples, the privacy-protective data management controller determines a number of feature points of the first set of feature points that moves in the first direction and a number of feature points of the second set of feature points that move in the non-uniform directions. A set of feature points that is associated with the background of an image (i.e., moves uniformly in an opposite direction from the camera) often has a greater number of feature points than a set of feature points that is associated with the foreground of the image (i.e., moves in various non-uniform directions and lengths). In other words, at <NUM>, when the number of feature points of the first set of feature points is greater than the number of feature points of the second set of feature points, the privacy-protective data management controller determines that the first set of feature points is associated with the background and the second set of feature points is associated with one or more foreground objects. Additionally, because the movement of the first set of feature points is substantially uniform and the movement of the second set of feature points is substantially non-uniform, the privacy-protective data management controller identifies the first set of feature points as background and the second set of feature points as foreground objects. At <NUM>, the privacy-protective data management controller alters the ROIs associated with each feature point from the second set of feature points to produce an altered image to protect privacy-related information included in the region of interest associated with each feature point from the second set of feature points. At <NUM>, the privacy-protective data management controller sends the altered image to the communications interface or store the altered image in a standardized format. While described in <FIG> as being associated with a moving camera, in other examples, the technique shown and described with respect to <FIG> can be used with images received from a non-moving camera.

<FIG> is a flow chart illustrating a combined privacy-protective data management method <NUM>, according to an embodiment. The combined privacy-protective data management method <NUM> can be executed at, for example, a privacy-protective data management controller such as the privacy-protective data management controller <NUM> shown and described with respect to <FIG> or the privacy-protective data management controller <NUM> shown and described with respect to <FIG>. More specifically, the combined privacy-protective data management method <NUM> can be executed by a processor, such as processor <NUM> of privacy-protective data management controller <NUM> shown and described with respect to <FIG>. In some embodiments, the privacy-protective data management controller can combine at least two of the optical flow-based non-claimed example illustrated with regards to <FIG>, the geometry-based embodiment illustrated with regards to <FIG>, and/or an object detector example not covered by the claims to improve the reliability of the privacy protection in different use cases.

For example, the privacy-protective data management controller receives a video or a set of images associated with a video recorded by a moving camera, at <NUM>. The video (or a set of images associated with a video), recorded by a moving camera (not shown), can include non-moving and structured buildings, a set of structured cars moving in different directions, and a person of interest moving in a direction. The moving camera can be on a moving drone configured to record activities of the person of interest.

At <NUM>, the privacy-protective data management controller detects, based on a characteristic associated with a person, a face, or an object, a first region of interest from the set of regions of interest and associate with an identity of a person. For example, the privacy-protective data management controller can define a machine learning model and train the machine learning model by manually marking certain ROIs in a set of sample images as a person, a face, or an object, and optimize a set of parameters associated with the machine learning model. Based on the machine learning model and the set of parameters, the privacy-protective data management controller can automatically detect certain features in an image as a person, a face, or an object. Once detected, at <NUM>, the privacy-protective data management controller tracks such features in the set of images of the video and alter these feature to produce an altered set of images.

At <NUM>, the privacy-protective data management controller defines a first moving direction of each feature point of the first set of feature points, and a second moving direction of each feature point from the second set of feature points. In some situations, a first subset of feature points from the set of feature points moves in a first direction, and a second subset of feature points from the set of feature points moves in a second direction different from the first direction. In some implementations, the privacy-protective data management controller determines a number of feature points of the first set of feature points that moves in the first direction and a number of feature points of the second set of feature points that move in the non-uniform directions. A set of feature points that is associated with the background of an image (i.e., moves uniformly in an opposite direction from the camera) often has a greater number of feature points than a set of feature points that is associated with the foreground of the image (i.e., moves in various non-uniform directions and lengths). In other words, when the number of feature points of the first set of feature points is greater than the number of feature points of the second set of feature points, the privacy-protective data management controller determines that the first set of feature points is associated with the background and the second set of feature points is associated with one or more foreground objects. Additionally, because the movement of the first set of feature points is substantially uniform and the movement of the second set of feature points is substantially non-uniform, the privacy-protective data management controller identifies the first set of feature points as background and the second set of feature points as foreground objects.

At <NUM>, the privacy-protective data management controller detects a structure (or edges, textures) of a region of interest from the set of regions of interest in the image. Background structures such as buildings, cars, roads, and/or the like differ in their geometry to the structure of humans and animals. For example, background structures such as buildings, cars, roads, and/or the like often have substantially straight lines, while the structure of humans and animals often does not have substantially straight lines. The privacy-protective data management controller classifies the structure of each region of interest into a geometric class from a set of predefined geometric classes and detects a fourth region of interest within the set of images and associated with the identity of a person (or an object).

At <NUM>, the privacy-protective data management controller then alters (or blur, scramble, pixelate, blank) the first ROI, the second ROI, and/or the fourth ROI to remove any privacy-related information in the ROIs and generate an altered image, such that privacy associated with the identity of the person is protected. At <NUM>, the privacy-protective data management controller sends the altered image to a user interface or store the altered image in a standardized format. While described in <FIG> as being associated with a moving camera, in other embodiments, the technique shown and described with respect to <FIG> can be used with images received from a non-moving camera.

In some not claimed examples, an apparatus includes a system to automatically or semi-automatically protect privacy while managing data generated with moving cameras. This can include (<NUM>) automatic detection of privacy-related information in the input data such as regions of interest (ROIs) in image frames, audio channels, or particular fragments of metadata, which might be relevant to revealing the identity of a person (e.g., face, body, license plate, color of clothing, voice, GPS coordinates, date and time), (<NUM>) a user interface, which allows a user to confirm or to correct these detections and to select additional fragments of information, which can be privacy protected, and (<NUM>) privacy protection of the information selected in (<NUM>) and (<NUM>).

According to some not claimed examples, the data management system includes a module that examines data integrity by inspecting the input data and searching for indications of tampering. Indexing of the data can be carried out in some embodiments to facilitate quick subsequent search at later use of the data. Another embodiment may include a system module, which provides a consistent output format and performs data compression to optimize the size of data to be stored.

Some not claimed examples described herein relate to a computer system as depicted in <FIG>, wherein privacy protector <NUM> automatically or semi-automatically selects privacy-related information in data generated with moving cameras and provides privacy protection to persons by scrambling the selected information fragments. The privacy protector <NUM> is structurally and functionally similar to the privacy-protective data management controller described with regards to <FIG> and <FIG>. Apart from its main privacy protecting functionality, in some embodiments the system can provide additional data management services such as integrity check (integrity checker <NUM>), indexing (indexer <NUM>), data compression and format uniformization (compressor/encoder <NUM>).

Metadata generated through the apparatus can further be used for other functionality such as search on metadata, behavior extraction and general description extraction (height estimates, clothing color estimates). Further, events and/or alerts can be defined and generated automatically based on this metadata.

In some embodiments, the privacy protector <NUM> can be hardware and/or software (implemented in hardware such as a processor). A system implementing the privacy protector <NUM> can include a processor and a memory storing code representing instructions to execute the privacy protector <NUM>. Additional modules and/or components described herein can include code stored in the memory be executed by the processor.

Some embodiments described herein relate to a system described above where privacy protection is carried out either in substantially real-time simultaneously while data is being captured or posteriorly on stored data. Some not claimed embodiments described herein automatically select privacy-related visual information in videos from moving cameras by separating background and foreground regions (e.g., as described above) in auto selector <NUM>. For example, auto selector <NUM> can receive video data associated with one or more images, can separate the background and the foreground, can identify privacy-related information (e.g., data and/or portions of the image to protect and/or obscure) in the foreground and can output the data associated with the separated background and foreground and/or the privacy-related information.

In some instances, foreground regions contain privacy sensitive information. Accurate separation of background and foreground regions can be a challenging task in case of moving cameras, because the three-dimensional (3D) structure of the scene is reconstructed based on the two-dimensional (2D) input data to distinguish between image pixels that are moving due to camera motion and pixels that are actually moving relative to the scene background (i.e. belonging to independently moving objects).

Some embodiments described herein use information about the scene geometry in the image frames of videos to do the above-mentioned background/foreground separation. One not claimed method of obtaining this information is as follows.

Some not claimed examples embodiments described herein can further enhance foreground extraction by taking advantage of the parallax effect (i.e., that objects close to the camera appear to be moving faster than objects farther away from the camera). Some embodiments described herein can use object or person detectors of auto selector <NUM> to fine-tune the selection of privacy sensitive image regions based on foreground extraction. Some embodiments described herein can further enhance the accuracy of privacy-related region selection with the help of tracking detected objects, persons, or faces using auto selector <NUM>.

Some not claimed examples described herein can use additional data in auto selector <NUM> collected by auxiliary sensors of a moving camera to automatically select privacy-related information in the input data. For example, the depth information from an infrared sensor of an Red Green Blue Depth ("RGBD") camera may significantly boost the accuracy of foreground extraction.

Some not claimed examples described herein can automatically select privacy-related information in the audio channels of the input data by using diarization in auto selector <NUM>. Some embodiments described herein automatically select privacy-related information in the audio channels of the input data by using median filtering and factorization techniques in auto selector <NUM>. Some embodiments described herein automatically select privacy-related information in the audio channels of the input data by using machine learning techniques in auto selector <NUM> (as described above).

Some not claimed examples described herein automatically select privacy-related information in the input metadata by applying artificial intelligence methods in auto selector <NUM>. The artificial intelligence agent selects fragments of the metadata based on their privacy-sensitivity in the given scenario.

Some not claimed examples described herein automatically select privacy-related information in the input metadata by using data mining in auto selector <NUM>. Some embodiments described herein automatically select privacy-related information in the input data in auto selector <NUM> and provide the opportunity to a user to manually modify the selection using manual selector <NUM> through the user interface <NUM>. In some scenarios, certain modifications can be made in the automatic selection of privacy-related information provided by auto selector <NUM>. These modifications can be correction, addition, or removal of privacy-related information fragments.

For example, the manual selector <NUM> can receive privacy-related information from the auto selector <NUM>. The manual selector <NUM> can present the privacy-related information (and the remaining portions of an image or video) to a user via the user interface <NUM>. This can allow a user to select other portions of the video and/or image to protect and/or obscure. The manual selector <NUM> can then receive a signal from the user interface <NUM> indicating other portions of the video and/or image to protect and/or obscure. The information associated with the portions of the video and/or image to protect and/or obscure can then be sent to the scrambler <NUM>.

Some embodiments described herein protect privacy by scrambling video data in scrambler <NUM>. For example, the scrambler <NUM> can receive the information associated with the portions of the video and/or image to protect and/or obscure from the manual selector <NUM> (or the auto selector <NUM>). The scrambler <NUM> can then process the portions of the video and/or image to protect and/or obscure such portions, as described herein. The scrambler <NUM> can then send the modified video and/or image (or image data) to the compressor/encoder <NUM>.

In some not claimed examples, the scrambler <NUM> can protect privacy by scrambling video data by blurring ROIs previously selected by auto selector <NUM> and/or manual selector <NUM> or whole image frames. Some implementations described herein protect privacy by scrambling video data in scrambler <NUM> by pixelating ROIs previously selected by auto selector <NUM> and/or manual selector <NUM> or whole image frames. Some implementations described herein protect privacy by scrambling video data in scrambler <NUM> by blanking ROIs previously selected by auto selector <NUM> and/or manual selector <NUM> or whole image frames. Some implementations described herein protect privacy by scrambling video data in scrambler <NUM> by in-painting ROIs previously selected by auto selector <NUM> and/or manual selector <NUM> or whole image frames.

Some not claimed examples described herein protect privacy by scrambling video data in scrambler <NUM> by replacing objects and people with their silhouettes in ROIs previously selected by auto selector <NUM> and/or manual selector <NUM> or in whole image frames. Some implementations described herein protect privacy by scrambling video data in scrambler <NUM> by applying JPEG-masking to ROIs previously selected by auto selector <NUM> and/or manual selector <NUM> or to whole image frames. Some implementations described herein protect privacy by scrambling video data in scrambler <NUM> by warping ROIs previously selected by auto selector <NUM> and/or manual selector <NUM> or whole image frames. Some implementations described herein protect privacy by scrambling video data in scrambler <NUM> by morphing ROIs previously selected by auto selector <NUM> and/or manual selector <NUM> or whole image frames. Some implementations described herein protect privacy by scrambling video data in scrambler <NUM> by cartooning ROIs previously selected by auto selector <NUM> and/or manual selector <NUM> or whole image frames.

Some not claimed examples described herein protect privacy by scrambling video data in scrambler <NUM> by applying various computer vision or image/video processing methods to ROIs previously selected by auto selector <NUM> and/or manual selector <NUM> or to whole image frames. Some implementations described herein protect privacy by scrambling previously selected fragments of audio data in scrambler <NUM> by applying various audio effects such as pitch shifting, etc. Some implementations described herein protect privacy by scrambling or erasing previously selected fragments of metadata in scrambler <NUM>.

According to some not claimed examples, privacy protector <NUM> can output multiple data streams providing various privacy-levels depending on the target utilization of the output data. For example, if the output data is planned to be used publicly, each and every person's privacy can be protected; if it is used as a forensic evidence, only the privacy of persons not involved in the incident is protected; users authorized having full access to the raw data can obtain unprotected output. The definition of different privacy-levels can be role and scenario dependent.

Some not claimed examples described herein can include integrity checker <NUM>, which performs an integrity check on the input data to determine whether the data has been tampered with. This integrity check can be performed, for example, by examining time stamps, correctness of video encoding, pixel-relations in video, and/or the like. For example, the integrity checker <NUM> can receive input data, can identify a source, time, checksum and/or other characteristics associated with the input data. This can be compared against an indication of expected input data to determine whether the data has been modified. The integrity checker can then provide the input data to the indexer <NUM>. In some instances, if an error is found by the integrity checker <NUM>, the integrity checker <NUM> can send an indication to a user (e.g., over a network and/or using the user interface <NUM>).

Some not claimed examples described herein can include indexer <NUM>, which performs indexing on the input data thereby generating additional metadata based on the content, which helps in conducting subsequent searches. This metadata can be for instance an activity index based on difference images, selected key-frames based on event detection, a textual representation of visual data, and/or the like. For example, the indexer <NUM> can receive data from the integrity checker <NUM>, can perform the indexing on the data and generate the additional metadata, and can send the data and the additional metadata to the privacy protector <NUM>.

Some not claimed examples described herein can include compressor/encoder <NUM>, which provides effective data compression and encodes output data in a consistent format. For example, the compressor/encoder <NUM> can receive protected data from the privacy protector <NUM> and can compress and/or encode the data. The compressor/encoder <NUM> can then send the data (e.g., output data) to a destination device (e.g., database, server, user interface, etc.), for example, over a network.

According to some implementations, compressor/encoder <NUM> can use audio coding formats such as MPEG-<NUM> or MPEG-<NUM> Audio Layer III, Vorbis, Windows Media Audio, Pulse-code modulation, Apple Lossless Audio Codec, Free Lossless Audio Codec, Money's Audio, OptimFROG, Tom's verlustfreier Audiokompressor, WavPack, TrueAudio, Windows Media Audio Lossless, and/or the like. According to some implementations, compressor/encoder <NUM> can use video coding formats such as Apple QuickTime RLE, Dirac lossless, FastCodec, FFV1, H. <NUM> lossless, JPEG <NUM> lossless, VP9, Cinepak, Dirac, H. <NUM>, MPEG-<NUM> Part <NUM>, H. <NUM>/MPEG-<NUM> Part <NUM>, H. <NUM>, MPEG-<NUM> Part <NUM>, H. <NUM>/MPEG-<NUM> AVC, HEVC, RealVideo, Theora, VC-<NUM>, Windows Media Video, MJPEG, JPEG <NUM>, DV, H. <NUM>, and/or the like.

While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. Where methods and/or schematics described above indicate certain events and/or flow patterns occurring in certain order, the ordering of certain events and/or flow patterns may be modified. While the embodiments have been particularly shown and described, it will be understood that various changes in form and details may be made within the scope of the independent claims.

For example, some not claimed examples can perform indexing (indexer <NUM>) before the integrity check (integrity checker <NUM>), do not include some components and/or modules such as integrity checker <NUM>, indexer <NUM>, and/or compressor/encoder <NUM>, and/or do not provide manual modification possibilities through manual selector <NUM> and/or user interface <NUM>, but function in a fully automatic way.

In some not claimed examples, privacy protection can be performed on information fragments, objects of interest, and/or other portions of an image and/or video automatically selected. In such implementations, the privacy can be protected by blurring selected ROIs or whole image frames of the video data; pixelating selected ROIs or whole image frames of the video data; blanking selected ROIs or whole image frames of the video data; in-painting selected ROIs or whole image frames of the video data; replacing objects and people with their silhouettes in selected ROIs or whole image frames of the video data; applying JPEG-masking to selected ROIs or whole image frames of the video data; warping selected ROIs or whole image frames of the video data; morphing selected ROIs or whole image frames of the video data; cartooning selected ROIs or whole image frames of the video data; applying various computer vision or image/video processing methods to selected ROIs or whole image frames of the video data; and/or the like.

In some not claimed examples, privacy-related information is automatically selected in audio data by using diarization, median filtering and factorisation techniques, machine learning techniques, and/or the like. In some not claimed examples, privacy is protected by distorting selected parts of the audio data by applying various audio effects such as, for example, pitch shifting and/or the like. In some implementations, privacy-related information is automatically selected in metadata by applying artificial intelligence methods, using data mining and/or the like. In some not claimed examples, privacy is protected by scrambling or erasing selected fragments of the metadata.

In some not claimed examples, privacy-related information is manually selected in the input data by a user through a user interface. In some not claimed examples, the automatic selection of privacy-related information is manually modified by a user through a user interface.

In some not claimed examples, integrity of input data is exampled by, for example, examining time-stamps, correctness of video encoding, or pixel-relations in video, and/or the like. In some not claimed examples, the input data is processed by an indexer in order to help subsequent search. Such indexing can be carried out by, for example, selecting key-frames, generating an activity index based on optical flow, providing a textual representation of visual data based on some artificial intelligence, and/or the like. In some implementations, the output data can be compressed in an optimized manner and encoded in a consistent format.

Due to the possibilities offered by modern technology, an enormous amount of data circulate extremely fast around the globe particularly through video sharing services and social networks on the Internet where camera generated data also appears frequently. The original source and the lifecycle of the data are mostly unknown, which makes the data unreliable. When the data from moving cameras is used, for example, as forensic evidence, this unreliability and the lack of data integrity is especially critical. Thus, there is a need for methods determining whether data integrity is ensured from capturing the data with a moving camera until its consumption.

Moreover, in certain situations, quickly identifying particular information in the data generated by moving cameras can be beneficial. To support such a fast search, the data management system can include a module that performs indexing on the input data. This indexing can be carried out through an arbitrary method, which is able to generate an abstract representation of the input data in which searching is more efficient and user friendly.

The inconsistent use of various data formats and non-optimal data compression may lead to serious issues while using and storing the data generated with moving cameras. Therefore, the data management system can include a module that provides the output in a consistent format and in optimal size.

The privacy-protective data management controller can be configured to include a manual (fine-tuning, tweaking) component, which can use the manual input as a feedback/training data for the automatic algorithm. The automatic algorithm can have a machine learning module in it thereby adapting its settings to the current scene/scenario.

Some embodiments described herein relate to a computer storage product with a non-transitory computer-readable medium (also can be referred to as a non-transitory processor-readable medium) having instructions or computer code thereon for performing various computer-implemented operations. The computer-readable medium (or processor-readable medium) is non-transitory in the sense that it does not include transitory propagating signals per se (e.g., a propagating electromagnetic wave carrying information on a transmission medium such as space or a cable). The media and computer code (also can be referred to as code) may be those designed and constructed for the specific purpose or purposes. Examples of non-transitory computer-readable media include, but are not limited to, magnetic storage media such as hard disks, floppy disks, and magnetic tape; optical storage media such as Compact Disc/Digital Video Discs (CD/DVDs), Compact Disc-Read Only Memories (CD-ROMs), and holographic devices; magneto-optical storage media such as optical disks; carrier wave signal processing modules; and hardware devices that are specially configured to store and execute program code, such as Application-Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), Read-Only Memory (ROM) and Random-Access Memory (RAM) devices. Other embodiments described herein relate to a computer program product, which can include, for example, the instructions and/or computer code discussed herein.

Claim 1:
An apparatus (<NUM>), comprising:
a memory; and
a processor operatively coupled to the memory, the processor configured to:
receive a set of images (<NUM>(<NUM>)-<NUM>(n)) associated with a video recorded by a moving or a non-moving camera,
detect a structure of a region of interest from a plurality of regions of interest (<NUM>-<NUM>) in an image from the set of images,
classify the structure into a geometric class from a set of predefined geometric classes defined using machine learning techniques, based on a machine learning model and a set of parameters stored in the memory,
alter the region of interest to generate an altered image when the geometric class is associated with an identity of a person, such that privacy associated with the identity of the person is protected, and
at least one of send the altered image to a user interface or store the altered image in a standardized format,
wherein the geometric class is associated with a background and not associated with the identity of the person when the structure of the region of interest includes a set of substantially straight lines,
wherein the processor is further configured to:
define a grid placed over the set of images, the grid including a plurality of grid cells (<NUM>-<NUM>), each grid cell from the plurality of grid cells associated with a different region of interest from the plurality of regions of interest in the image,
detect a structure of each region of interest in each grid cell from the plurality of grid cells, and
classify each grid cell into the set of predefined geometric classes using the set of parameters stored in the memory.