METHOD AND APPARATUS FOR CANCELLING ANONYMIZATION FOR AN AREA INCLUDING A TARGET

In an aspect, there is provided method for cancelling anonymization for an area including a target in an input image, the input image including a plurality of identified subjects, the method comprising: retrieving historical images in which the target has appeared in; identifying an area by comparing the input image and the retrieved historical images to determine if the target has interacted with at least one of the identified subjects; and cancelling anonymization for the area including the target.

TECHNICAL FIELD

The present invention relates broadly, but not exclusively, to a method and an apparatus for cancelling anonymization for an area including a target and their interacted subjects.

BACKGROUND ART

Video surveillance is widely used by law and security enforcement as the primary tool to monitor their areas and prevent crime and terrorism. Though the general public support the usage of video surveillance because of their effectiveness in crime-prevention, deterrence, and post investigation, there is also growing privacy concerns of their misuse and abuse as they do involve intruding into ordinary people everyday lives.

The video data captured contain privacy sensitive information and this indiscriminate shooting is a critical issue for people who highly regard their privacy and are unwilling to be revealed.

Privacy infringement can become a significant social problem. To tackle privacy infringement, video surveillance needs to anonymize irrelevant and unnecessary data without forgoing its capabilities and performance.

SUMMARY OF INVENTION

Technical Problem

Existing face recognition solutions often apply anonymization technologies on the detected face for data privacy protection. Similar to all other object detection algorithms limitation, face detection algorithms in face recognition solution may face missed detection due to algorithms configuration and training dataset.

Solution to Problem

The present provides a method for cancelling anonymization for an area including a target in an input image, the method comprising: retrieving historical images in which the target has appeared in; identifying an area by comparing the input image and the retrieved historical images to determine if the target has interacted with at least one of the identified subjects; and cancelling anonymization for the target and identified subjects that the target has interacted with.

The present provides an apparatus for cancelling anonymization for an area including a target in an input image, the apparatus comprising: at least one memory that is configured to store instructions; and at least one processor that is configured to execute the instructions to: retrieve historical images in which the target has appeared in; identify an area by comparing the input image and the retrieved historical images to determine if the target has interacted with at least one of the identified subjects; and cancel anonymization for the area including the target.

DESCRIPTION OF EMBODIMENTS

Herein disclosed are embodiments of an apparatus and methods for cancelling anonymization for an area including a target that addresses one or more of the above problems.

Furthermore, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure.

Terms Description

Subject—a subject may be any suitable type of entity, which may include a person, a patient and a user.

The term target or target subject is used herein to identify a person, a user or patient that is of interest. The target subject may be one that is selected by a user input or one who is identified to be of interest.

A subject or an identified subject is used herein to relate to a person who is related to or appear with the target subject (e.g., partner or companion) in an input image. For example, in the context of pandemic outbreak, the subject is someone who may have interacted or appeared together with the target.

A user who is registered to a contact tracing server will be called a registered user. A user who is not registered to the contact tracing server will be called a non-registered user. It is possible for the user to obtain graphical representation of any subject on a network graph.

Contact tracing server—The contact tracing server is a server that hosts software application programs for receiving inputs, processing data and objectively providing graphical representation. The contact tracing server communicates with any other servers (e.g., a remote assistance server) to manage requests. The contact tracing server communicates with a remote assistance server to display a graphical representation of a potential subject and a target subject. Contact tracing servers may use a variety of different protocols and procedures in order to manage the data and provide a graphical representation.

The contact tracing server is usually managed by a provider that may be an entity (e.g., a company or organization) which operates to process requests, manage data and display graphical representations that are useful to a situation. The server may include one or more computing devices that are used for processing graphical representation requests and providing customizable services depending on situations.

A contact tracing account—a contact tracing account is an account of a user who is registered at a contact tracing server. In certain circumstances, the contact tracing account is not required to use the remote assistance server. A contact tracing account includes details (e.g., name, address, vehicle etc.) of a user.

The contact tracing server manages contact tracing accounts of users and the interactions between users and other external servers, along with the data that is exchanged.

DETAILED DESCRIPTION

It is to be noted that the discussions contained in the “Background” section and that above relating to prior art arrangements relate to discussions of devices which form public knowledge through their use. Such should not be interpreted as a representation by the present inventor(s) or the patent applicant that such devices in any way form part of the common general knowledge in the art.

The System100

FIG.1illustrates a block diagram of a system100for cancelling anonymization for areas of the target person and areas of any identified subjects interacted with the target person in the historical images. The system100comprises a requestor device102, a contact tracing server108, a remote assistance server140, remote assistance hosts150A to150N, and sensors142A to142N.

The requestor device102is in communication with a contact tracing server108and/or a remote assistance server140via a connection116and121, respectively. The connection116and121may be wireless (e.g., via NFC communication, Bluetooth, etc.) or over a network (e.g., the Internet). The connection116and121may also be that of a network (e.g., the Internet).

The contact tracing server108is further in communication with the remote assistance server140via a connection120. The connection120may also be that of a network (e.g., the Internet). In one arrangement, the contact tracing server108and the remote assistance server140are combined and the connection120may be an interconnected bus.

The remote assistance server140, in turn, is in communication with the remote assistance hosts150A to150N via respective connections122A to122N. The connections122A to122N may be a network (e.g., the Internet).

The remote assistance hosts150A to150N are servers. The term host is used herein to differentiate between the remote assistance hosts150A to150N and the remote assistance server140. The remote assistance hosts150A to150N are collectively referred to herein as the remote assistance hosts150, while the remote assistance host150refers to one of the remote assistance hosts150. The remote assistance hosts150may be combined with the remote assistance server140.

In an example, the remote assistance host150may be one managed by a hospital and the remote assistance server140is a central server that manages emergency calls and decides which of the remote assistance hosts150to forward data or retrieve data like image inputs.

Sensors142A to142N are connected to the remote assistance server140or the contact tracing server108via respective connections144A to144N or144A to144N. The sensors142A to142N are collectively referred to herein as the sensors142A to142N. The connections144A to144N are collectively referred to herein as the connections144, while the connection144refers to one of the connections144. Similarly, the connections146A to146N are collectively referred to herein as the connections146, while the connection146refers to one of the connections146. The connections144and146may also be that of a network. The sensor142may be one of an image capturing device, video capturing device, and motion sensor and may be configured to send an input depending its type, to at least one of the contact tracing server108.

In the illustrative embodiment, each of the devices102and142; and the servers108,140, and150provides an interface to enable communication with other connected devices102and142and/or servers108,140, and150. Such communication is facilitated by an application programming interface (“API”). Such APIs may be part of a user interface that may include graphical user interfaces (GUIs), Web-based interfaces, programmatic interfaces such as application programming interfaces (APIs) and/or sets of remote procedure calls (RPCs) corresponding to interface elements, messaging interfaces in which the interface elements correspond to messages of a communication protocol, and/or suitable combinations thereof.

Use of the term ‘server’ herein can mean a single computing device or a plurality of interconnected computing devices which operate together to perform a particular function. That is, the server may be contained within a single hardware unit or be distributed among several or many different hardware units.

The Remote Assistance Server140

The remote assistance server140is associated with an entity (e.g., a company or organization or moderator of the service). In one arrangement, the remote assistance server140is owned and operated by the entity operating the server108. In such an arrangement, the remote assistance server140may be implemented as a part (e.g., a computer program module, a computing device, etc.) of server108.

The remote assistance server140may also be configured to manage the registration of users. A registered user has a contact tracing account (see the discussion above) which includes details of the user. The registration step is called on-boarding. A user may use either the requestor device102to perform on-boarding to the remote assistance server140.

It is not necessary to have a contact tracing account at the remote assistance server140to access the functionalities of the remote assistance server140. However, there are functions that are available to a registered user. For example, it may be possible to display graphical representation of target subjects and potential subjects in other jurisdictions. These additional functions will be discussed below.

The on-boarding process for a user is performed by the user through one of the requestor device102. In one arrangement, the user downloads an app (which includes the API to interact with the remote assistance server140) to the sensor142. In another arrangement, the user accesses a website (which includes the API to interact with the remote assistance server140) on the requestor device102.

Details of the registration include, for example, name of the user, address of the user, emergency contact, or other important information and the sensor142that is authorized to update the remote assistance account, and the like.

Once on-boarded, the user would have a contact tracing account that stores all the details.

The Requestor Device102

The requestor device102is associated with a subject (or requestor) who is a party to a contact tracing request that starts at the requestor device102. The requestor may be a concerned member of the public who is assisting to get data necessary to obtain a graphical representation of a network graph. The requestor device102may be a computing device such as a desktop computer, an interactive voice response (IVR) system, a smartphone, a laptop computer, a personal digital assistant computer (PDA), a mobile computer, a tablet computer, and the like.

In one example arrangement, the requestor device102is a computing device in a watch or similar wearable and is fitted with a wireless communications interface.

The Contact Tracing Server108

The contact tracing server108is as described above in the terms description section.

The contact tracing server108is configured to process processes relating to cancel anonymization for areas of the target person and areas of any identified subjects in the historical images.

The Remote Assistance Hosts150

The remote assistance host150is a server associated with an entity (e.g., a company or organization) which manages (e.g., establishes, administers) information regarding information relating to a subject.

In one arrangement, the entity is a hospital. Therefore, each entity operates a remote access host150to manage the resources by that entity. In one arrangement, a remote access host150receives an alert signal that a target subject is likely to be carrier of a virus.

Advantageously, such information is valuable to the law enforcement. It reduces number of hours looking through camera footage to investigate the possible links between persons of interest who may have interacted in the past.

The information is particularly useful in the pandemic, so that building management and the health sector can more efficiently and effectively carry out contact tracing. The network of interactions between the target and other subjects helps to identify how the disease spreads from one person to another.

Conventionally, the relevant subjects who may have interacted with the target is not shown. That is, anonymization is only cancelled for the target. There are also other conventional techniques that anonymize using detection technique. That is, all detected persons in an input image are anonymized. In an existing technique, moving subject are anonymized using foreground or background extraction. More commonly, background of an image may be blurred or image anonymization may be done on a predefined segment/area of image, or the other way around, to exclude the area of interest of image anonymization.

The present disclosure provides image anonymization to protect innocent person but cancel anonymization for targeted person with his/her interacted potential subjects. Combining detected face algorithms of targeted person with full image background anonymization can helps to preserve better privacy of innocent person, but it also hides away many details of interaction of targeted person with his/her potential associates around him/her.

The sensor142is associated with a user associated with the requestor device102. More details of how the sensor may be utilized will be provided below.

FIG.2Ashows how conventional technique cancels anonymization for the target. As shown inFIG.2A, conventional technique cancels anonymization for the target202. Subject204may be one who has interacted with the target202and according to conventional technique, anonymization is not cancelled for subject204.

FIG.2Bshows how the present disclosure anonymization for the target and an identified subject. As shown inFIG.2B, this present disclosure cancels anonymization for the target202and subject204who may be one who has interacted with the target202. More information can be provided in the following.

FIG.3shows the key advantages according to an embodiment of the present disclosure. An input image302including a target202and a subject204among a plurality of other subjects may be received. At304, image anonymization is performed based on configured or predefined algorithms. At step306, all pre-stored and labelled target person's historical images are retrieved from a database. At step308, a processing step is performed to detect the interaction by the target with other subjections by calculating intersection between rectangle coordinates of two people for retrieved historical information. At step310, the area in the input image302involving the target and subjects who have interacted with the target are de-anonymized. An output image312is output.

FIG.4shows a step of how the present disclosure cancels anonymization for the target and an identified subject according to an embodiment of the present disclosure.

InFIG.4, an input image402is received. The method includes applying background image subtraction to input image402for foreground image extraction as shown in404. A sample extracted foreground image is shown in406.

FIG.5may followFIG.4. As shown in502, face recognition may be performed on the extracted foreground image against a predefined target list. As a result of the face recognition, #4 may be matched as the target person found in targeted person list.

FIG.6may followFIG.5. InFIG.6, retrieve all the pre-stored historical images of the matched target are retrieved as shown in602. That is, historical information relating to the target is retrieved and images in which the target have appeared are identified, as shown in604.

FIG.7may followFIG.6. InFIG.7, subjects, for example, #1, #2 and #3, who have appeared with the target in the input image are identified. It is to be appreciated that it may be just one subject in some embodiments. In702, face matching for each identified subjects, are performed on the retrieved images of the targeted (#4). #1 and #3 subject from the input image are detected in the retrieved historical information of #4 target. #1 appeared in the retrieved historical information of #4 target twice and #3 appeared in the retrieved historical information of #4 target once. As such, the count of co-appearances for #4 and #1 is 2; that for #4 and #2 is 0 and that #4 and #3 is 1.

FIG.8may followFIG.7. The count of interaction for #4 and #1 is 2; that for #4 and #2 is 0 and that #4 and #3 is 1 based on the historical information is shown in802.

In804, rectangle coordinates in pixel for each subject is calculated in image segmentation, where X=min X, left most position; Y=min Y, top most position; H=max height of image segment; W=max weight of image segment. In an embodiment, the rectangle coordinates may be calculated based on XY1={X, Y}; XY2={X+W, Y}; XY3={X, Y+H}; and XY4={X+W, Y+H}.

FIG.9may followFIG.8. As mentioned inFIG.7, face matching for each identified subjects, are performed on the retrieved images of the targeted (#4). #1 and #3 subject from the input image are detected in the retrieved historical information of #4 target. #1 appeared in the retrieved historical information of #4 target twice and #3 appeared in the retrieved historical information of #4 target once. As such, the count of co-appearances for #4 and #1 is 2; that for #4 and #2 is 0 and that #4 and #3 is 1. Image segmentation may be performed on each historical image and it may be determined that there is an intersection between the rectangular coordinates of #4 and #3 as shown in902. In this example, #3 and #4 may be de-anonymized in the input image.

FIG.10may followFIG.9.FIG.10shows in detail how rectangular coordinates may be used to determine if there is an intersection between a subject and the target subject. The following may be used:

Based on the Equations (1) and (2): Intersection is detected between A(R1) & B(R2):

Based on the Equations (1) and (2): Intersection is not detected between A(R1) & C(R2):

FIG.11shows a method of how to cancel anonymization for an area including a target according to an embodiment of the present disclosure. At step1102, the method starts. Input image include the target and the subjects are received at step1104. In an embodiment, pre-configures camera background may be obtained at step1106. At step1108, background subtraction is performed on the input image received at1102to get the foreground image. At step1110, face recognition is performed on the foreground image after the background subtraction. At step1112, the detected faces are matched against the predefined target list.

At step1114, it is determined if the target person is found. If it is determined that the target person is not found, the method proceeds to anonymize every subject in the input image in step1116. If it is determined that the target person is found, the method proceeds to retrieve all the pre-stored historical images of the matched target subject in step1118.

At step1120, it is determined if there are any historical images found. If it is determined that the historical images are not found, the method proceeds to anonymize every subject and reveal the found target person in the input image in step1132. If it is determined that the historical images are found, the method proceeds to perform face matching of all detected faces in the input image against all retrieved historical in step1122.

At step1124, it is determined if there are any interacted persons found. If it is determined that the interacted persons are not found, the method proceeds to anonymize every subject and reveal the target person in the input image in step1132. If it is determined that the interacted persons are found, the method proceeds to perform intersection detection on all co-appearances with found target subject identified in the input image in step1126.

At step1128, a list of detected subjects who have intersected with at least one target person is obtained. At step1130, the method anonymizes every subject and reveal the target person in the input image and the list of intersected persons in the input image. It then proceeds to the end of the method.

FIG.12shows how intersection between a subject and the target subject may be determined in an embodiment of the present disclosure. InFIG.12, three-dimensional (3D) distance between two persons is obtained by transforming the two-dimensional (2D) image person into real life 3D location and calculate the distance between two persons to determine if it's equal or less than a predefined distance, example within 1 meter.

It is assumed that the input f=35 mm, sensor=35 mm, height=170 cm, the distance of subject B from camera as shown in1202is tabulated to:

The distance of subject A from camera as shown in1202is tabulated to:

As such, it is tabulated that the distance between A and B from camera as shown in1204:

The distance between A and B as shown in1204is tabulated to:

Based on the tabulated distances between A and B from the camera, it is determined if the subjects A and B have intersected in an image.

Device1300, hereinafter interchangeably referred to as a computer system1300, where one or more such computing devices1300may be used to execute the methods shown above. The exemplary computing device1300can be used to implement the system100shown inFIG.1. The following description of the computing device1300is provided by way of example only and is not intended to be limiting.

As shown inFIG.13, the example computing device1300includes a processor1307for executing software routines. Although a single processor is shown for the sake of clarity, the computing device1300may also include a multi-processor system. The processor1307is connected to a communication infrastructure1306for communication with other components of the computing device1300. The communication infrastructure1306may include, for example, a communications bus, cross-bar, or network.

The computing device1300further includes a main memory1308, such as a random access memory (RAM), and a secondary memory1310. The secondary memory1310may include, for example, a storage drive1312, which may be a hard disk drive, a solid state drive or a hybrid drive and/or a removable storage drive1317, which may include a magnetic tape drive, an optical disk drive, a solid state storage drive (such as a USB flash drive, a flash memory device, a solid state drive or a memory card), or the like. The removable storage drive1317reads from and/or writes to a removable storage medium1377in a well-known manner. The removable storage medium1377may include magnetic tape, optical disk, non-volatile memory storage medium, or the like, which is read by and written to by removable storage drive1317. As will be appreciated by persons skilled in the relevant art(s), the removable storage medium1377includes a computer readable storage medium having stored therein computer executable program code instructions and/or data.

In an alternative implementation, the secondary memory1310may additionally or alternatively include other similar means for allowing computer programs or other instructions to be loaded into the computing device1300. Such means can include, for example, a removable storage unit1322and an interface1350. Examples of a removable storage unit1322and interface1350include a program cartridge and cartridge interface (such as that found in video game console devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a removable solid state storage drive (such as a USB flash drive, a flash memory device, a solid state drive or a memory card), and other removable storage units1322and interfaces1350which allow software and data to be transferred from the removable storage unit1322to the computer system1300.

The computing device1300also includes at least one communication interface1327. The communication interface1327allows software and data to be transferred between computing device1300and external devices via a communication path1327. In various embodiments of the inventions, the communication interface1327permits data to be transferred between the computing device1300and a data communication network, such as a public data or private data communication network. The communication interface1327may be used to exchange data between different computing devices600which such computing devices1300form part an interconnected computer network. Examples of a communication interface1327can include a modem, a network interface (such as an Ethernet card), a communication port (such as a serial, parallel, printer, GPIB, IEEE 1394, RJ45, USB), an antenna with associated circuitry and the like. The communication interface1327may be wired or may be wireless. Software and data transferred via the communication interface1327are in the form of signals which can be electronic, electromagnetic, optical or other signals capable of being received by communication interface1327. These signals are provided to the communication interface via the communication path1327.

As shown inFIG.13, the computing device1300further includes a display interface1302which performs operations for rendering images to an associated display1350and an audio interface1352for performing operations for playing audio content via associated speaker(s)1357.

As used herein, the term “computer program product” may refer, in part, to removable storage medium1377, removable storage unit1322, a hard disk installed in storage drive1312, or a carrier wave carrying software over communication path1327(wireless link or cable) to communication interface1327. Computer readable storage media refers to any non-transitory, non-volatile tangible storage medium that provides recorded instructions and/or data to the computing device1300for execution and/or processing. Examples of such storage media include magnetic tape, CD-ROM, DVD, Blu-ray™ Disc, a hard disk drive, a ROM or integrated circuit, a solid state storage drive (such as a USB flash drive, a flash memory device, a solid state drive or a memory card), a hybrid drive, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computing device1300. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and/or data to the computing device1300include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like.

The computer programs (also called computer program code) are stored in main memory1308and/or secondary memory1310. Computer programs can also be received via the communication interface1327. Such computer programs, when executed, enable the computing device1300to perform one or more features of embodiments discussed herein. In various embodiments, the computer programs, when executed, enable the processor1307to perform features of the above-described embodiments. Accordingly, such computer programs represent controllers of the computer system1300.

Software may be stored in a computer program product and loaded into the computing device1300using the removable storage drive1317, the storage drive1312, or the interface1350. The computer program product may be a non-transitory computer readable medium. Alternatively, the computer program product may be downloaded to the computer system1300over the communications path1327. The software, when executed by the processor1307, causes the computing device1300to perform the necessary operations to execute the method as described above.

It is to be understood that the embodiment ofFIG.13is presented merely by way of example to explain the operation and structure of the system100. Therefore, in some embodiments one or more features of the computing device1300may be omitted. Also, in some embodiments, one or more features of the computing device1300may be combined together. Additionally, in some embodiments, one or more features of the computing device1300may be split into one or more component parts.

This application is based upon and claims the benefit of priority from Singapore patent application No. 10202108597X, filed on Aug. 5, 2021, the disclosure of which is incorporated herein in its entirety by reference