Patent ID: 12197497

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention are explained below with reference to the drawings. However, the present invention is not limited to the following embodiments. It should be noted that in each diagram, the same reference signs are applied to the same members or elements, and duplicate descriptions will be omitted or simplified.

Hereinafter, a case in which a specific face is searched for as a subject will be explained as an example. A face of a search target is referred to as a query face, an image in which a query face is known to appear is referred to as a query image, and an image in which a query face may appear is referred to as an image to be searched. An object of the present invention is to search for a query face from an image to be searched.

However, the target to be searched of the present invention is not limited to a face, and can be any target for which identity can be determined by a feature vector. That is, in the embodiments, an example in which a face is a query subject will be explained, but a query subject is not limited to a face. For example, a query subject can be a specific animal, or a specific object.

First Embodiment

FIG.1is a diagram showing an example of a hardware configuration of an image processing apparatus according to a first embodiment. The image processing apparatus100includes a control device11, a storage device12, an input device13, an output device14, an OF device15, and the like serving as a hardware configuration.

The control device11incorporates a CPU serving as a computer and controls the entire image processing apparatus100. The storage device12holds computer programs and data necessary for the operation of the control device11. The input device13is a human interface device or the like, and inputs an operation of a user into the image processing apparatus100.

The output device14generates an image signal, such as a display image signal, to be displayed on a display device and the like (not shown), and presents the processing results of the image processing apparatus100or the like to the user. The OF device15is a wired interface such as a universal serial bus, an Ethernet (registered trademark), or an optical cable, or a wireless interface such as Wi-Fi or Bluetooth (registered trademark).

The OF device15includes a function for connecting a camera to input a captured image into the image processing apparatus100, a function for transmitting a processing result acquired by the image processing apparatus100externally, and a function for inputting a program and data and the like necessary for an operation of the image processing apparatus100into the image processing apparatus100and the like.

FIG.2is a functional block diagram showing an example of a functional configuration of the image processing apparatus in the First Embodiment. It should be noted that a portion of the functional blocks shown inFIG.2are realized by causing the CPU serving as a computer included in a control device to execute a computer program stored in a memory serving as a storage medium.

However, some or all of the functional blocks may be implemented by hardware. Serving as the hardware, a dedicated circuit (ASIC), a processor (a reconfigurable processor, or a DSP), or the like can be used.

Each functional block illustrated inFIG.2may not be embedded in the same casing or may be configured with separate devices connected via a signal path. The foregoing description inFIG.2similarly applies toFIGS.5,7,8, and10.

Serving as a functional configuration, the image processing apparatus100includes an image acquisition unit201, a feature vector extraction unit202, a feature vector map extraction unit203, a heat map calculation unit204, and a face position estimation unit205. In addition, the image processing apparatus100includes a feature extractor learning unit206and a feature vector map extractor learning unit207.

The image acquisition unit201serving as an image acquisition means acquires a query image, which is an image that includes a subject that is the search target, and an image to be searched, which may include the search target. A query image and an image to be searched acquired by the image acquisition unit201may be an image captured by a security camera or the like, may be an image recorded on a recording device such as a hard disk, or may be an image received via a network such as the Internet.

The query image that has been acquired by the image acquisition unit201is transmitted to the feature vector extraction unit202, and the image to be searched that has been acquired by the image acquisition unit201is transmitted to the feature vector map extraction unit203.

The feature vector extraction unit202detects a query face from the query image acquired by the image acquisition unit201, and extracts a feature vector of the query face. The feature vector that has been extracted by the feature vector extraction unit202is transmitted to the heat map calculation unit204. Here, the feature vector extraction unit202functions as a feature vector extraction means that extracts a first feature vector representing a feature of the search target that is included in the query image.

The feature vector map extraction unit203extracts a feature vector map from the image to be searched that was acquired by the image acquisition unit201. The feature vector map that has been extracted by the feature vector map extraction unit203is sent to the heat map calculation unit204. Here, the feature vector map extraction unit203functions as a feature vector map extraction means that extracts a feature vector map representing feature vectors of a subject at each position of the image to be searched.

The heat map calculation unit204calculates a heat map from the feature vectors extracted by the feature vector extraction unit202, and the feature vector map extracted by the feature vector map extraction unit203.

Here, the heat map calculation unit204functions as a heat map calculation unit that performs an arithmetic operation between the first feature vector, and the feature vector map, and calculates a heat map that represents the likelihood of the presence of a search target. The heat map that has been estimated by the heat map calculation unit204is sent to the face position estimation unit205.

The face position estimation unit205estimates a candidate position in which a query face may appear in the image to be searched based on the heat map that has been estimated by the heat map calculation unit204. Here, the face position estimation unit205functions as a candidate position estimation unit that estimates from the heat map candidate positions, which are positions in which a search target is most likely to appear in the image to be searched.

The feature extractor learning unit206trains the feature extractor that is used by the feature vector extraction unit202to extract feature vectors from an image, and stores the parameters thereof as feature extractor parameters208in the storage device12.

The feature vector map extractor learning unit207trains the feature vector map extractor that is used by the feature vector map extraction unit203to extract a feature vector map from the image. Then, the parameters thereof are stored as feature vector map extractor parameters209in the storage device12.

The image processing apparatus100according to the present embodiment trains a feature extractor and a feature vector map extractor during a training phase, and performs a search for a specific subject by using the trained feature extractor and the feature vector map extractor during a search phase. In the following, the flow of the training process and the flow of the search process will be explained in sequence.

An example of the flow of the training process of the image processing apparatus100according to the present embodiment will be explained with reference toFIG.3.FIG.3is a flowchart showing a flow of the training process by the image processing apparatus according to the First Embodiment. It should be noted that the operation of each step of the flowchart ofFIG.3is performed by a CPU that serves as a computer in a control device performing control by executing a computer program stored in a memory.

In step S301, the feature extractor learning unit206trains a feature extractor used by the feature vector extraction unit202to extract feature vectors from an image, and stores the parameters thereof as feature extractor parameters208in the storage device12. The feature extractor is trained by a known method such as one called distance learning, and the like.

A method of creating training data for use in training a feature extractor will be explained. First, a large number of face images with known person IDs are collected. The number of different person IDs in the training data is C, and the person IDs are assumed to be assigned sequentially from 1 to C.

Next, a normalized face image is acquired by applying a normalization process to a face image. The normalized face image is an image extracted so that both the height and width are a predetermined number of pixels, with no tilting of the face to the left or to the right, the size of the face is approximately constant, and the relative position of the face to the normalized face image is approximately constant. The purpose of the normalization process is to reduce variations in the in-plane rotational angle and size of a face captured in a face image.

FIG.4is a diagram showing an example of normalization of a face image. A face image400is an image that includes one of a face410with an ID (identifier) that is known as identifying information. A known organ point position estimation technique is used to estimate the positions of organ points such as the eyes, nose, and mouth of the face410in the face image400.

In a case in which position estimation of an organ point fails, the corresponding face image will not be included in the training data. InFIG.4, as an example, the positions of a right eye center401, a left eye center402, a right mouth end403, and a left mouth end404are estimated.

Next, a normalized face image405is extracted from the face image400based on the positions of the organ points. This extraction is performed, for example, by applying a geometric transformation, such as an affine transformation, to the face image400such that that the right eye center401approaches the predetermined position406, the left eye center402approaches the predetermined position407, the right mouth end403approaches the predetermined position408, and the left mouth end404approaches the predetermined position409of the normalized face image405. By the above-described method, training data can be created.

Next, the feature extractor is trained by using the created training data. The feature extractor is a deep neural network that takes a normalized face image as input, and outputs a feature vector of fixed dimensions. Hereinafter, the dimensionality of the feature vector is denoted as D. For example, D can take a value of 128, 256, 512, and the like.

The feature extractor can be configured by connecting a fully connected layer to a network that is derived from a Convolutional Neural Network (CNN), such as, for example, a Residual Neural Network (ResNet) or a Visual Geometry Group (VGG). In this case, the output of the fully connected layer is designed so as to be D.

During the training of the feature extractor, a class classification layer that takes the feature vector as input and outputs a C-dimensional probability vector is connected behind the feature extractor. An i-th component of the probability vector (where i is an integer from 1 to C) expresses the probability that the person ID of the face seen in the face image is i. The feature extractor and the class classification layer are trained by using a technique such as backpropagation so that the probability vector, which is the output of the class classification layer, approaches the One-Hot vector.

Here, the One-Hot vector refers to a vector in which, assuming that the ID of the correct person in the face image is denoted as j, only the j-th component is 1, and other components are 0. During the training of the feature extractor, a constraint is set so that the length (for example, the L2 norm) of the feature vector output by the feature extractor approaches a predetermined value.

The feature extractor is trained in advance according to the framework of distance learning. That is, the feature extractor is trained so as to output feature vectors at a short distance from each other for images of an identical subject or subjects belonging to an identical class. In addition, the feature extractor is trained in advance so as to output feature vectors at a long distance from each other for images of non-identical subjects or subjects that do not belong to an identical class.

For example, as in Deng et al, the feature extractor may be trained by a method that defines the angle between the feature vectors output by the feature extractor as a distance. That is, for images that include an identical subject or subjects belonging to an identical class, the feature extractor is trained so that the angle between the feature vectors output by the feature extractor becomes smaller than a predetermined angle. In addition, for images of non-identical objects or objects belonging to non-identical classes, the feature extractor is trained so that the angle between the feature vectors output by the feature extractor becomes larger.

A case is assumed in which an image that includes an identical subject or subjects belonging to an identical class is input to the feature vector extraction unit202and the feature vector map extraction unit203. In that case, the feature vector output by the feature vector extraction unit202may be trained in advance so that the difference in distance between the feature vector output by the feature vector extraction unit202and the feature vector at the position of the subject in the feature vector map becomes less than or equal to a predetermined value.

After training, by discarding the class classification layer, a feature extractor that extracts feature vectors from an image can be obtained. The feature extractor learning unit206stores the parameters of this feature extractor in the storage device12as feature extractor parameters208.

In step S302, the feature vector map extractor training unit207trains the feature vector map extractor used when the feature vector map extraction unit203extracts a feature vector map from the image. Then, the parameters thereof are stored as feature vector map extractor parameters209in the storage device12.

The feature vector map extractor is a deep neural network that takes an image having H for height and W for width as input, and outputs a feature vector map having a height of H/Kh, a width of W/Kw, and a depth of D.

Here, Khand Kware constants that determine the ratio between the size of the input image and the size of the feature vector map. The values of Khand Kwmay be set arbitrarily. As Khand Kware made larger, the size of the feature vector map becomes smaller, thereby reducing the computational complexity of the feature vector map extractor. Khand Kwmay be the same value.

The feature vector at the position (x/Kw, y/Kh) of the feature vector map represents the facial feature amount of a face in the vicinity of position (x, y) of the input image. This feature vector is designed such that the similarity between the feature vector and the feature vector that can be obtained when a face image with a face in the vicinity of the position (x, y) of the input image is input to the feature extractor trained in step S301becomes high. Hereinafter, an example of a training method for a feature vector map extractor will be explained.

First, a method of creating training data for use in training of a feature vector map extractor will be explained. The training data is configured from pairs of an image in which a face appears and the corresponding ground truth feature vector map for that image.

Hereinafter, an example of the creation of a ground truth feature vector map will be explained by usingFIG.5.FIG.5is a diagram showing an example of the creation of a ground truth feature vector map. First, an image501in which one or more faces appear is prepared. The height of the image501is H and width is W. Next, a vector map having a height of H/Kh, a width of W/Kw, and a depth of D is generated, and all values are initialized to zero.

Next, the positions of faces502to505are estimated from the image501. The position of a face can be acquired from annotation information separately provided to the image501, or by applying a known face detection technique to the image501.

Next, a normalized face image is generated for each of the faces502to505that are present in the image501by the same method that was explained in step S301, and a feature vector is estimated by applying the feature extractor that was trained in step S301. A normalized face image506is an example of a normalized face image corresponding to a face502. A feature vector507is an example of a feature vector corresponding to the face502.

Next, a feature vector corresponding to a face at a position (x, y) in the image501is added to a position (x/Kw, y/Kh) of the vector map. At this time, the feature vector may also be added to a position within a certain range from the position (x/Kw, y/Kh) of the vector map.

Furthermore, the feature vector may be added to the vector map after applying a process to shorten the length of the feature vector as the distance from the position (x/Kw, y/Kh) of the vector map increases.

For example, the center position of the face502in the image501is represented by (x1, y1), and the feature vector507corresponding to the face502is represented by v. The vector superimposed at the position (x1/Kw+Δx, y1/Kh+Δy) of the ground truth feature vector map becomes v·f(Δx, Δy)·g(Δx, Δy).

That is, the feature vector obtained when an image that includes the subject at the position (x, y) of the image is input to the feature vector extraction unit202is multiplied by a coefficient calculated based on Δx and Δy. Thereby, the feature vector at the position (x/Kw+Δx, y/Kh+Δy) of the ground truth feature vector map may be calculated.

Here, a function f (x, y) is a function that takes smaller values as x and y become larger (for example, a two-dimensional normal distribution such that the mean vector is (0, 0)T). In addition, a function g(x, y) is a function that takes the value 1 when Δx·Δx+Δy·Δy is less than or equal to the square of R, and takes the value 0 otherwise. Here, R is a predetermined value representing a real number other than 0.

By multiplying the function g(x, y), the feature vector map extractor is trained so as to output a zero vector corresponding to a non-face region in the image501. The vector map that has undergone the above-described processing becomes the ground truth vector map508corresponding to the image501.

The image501may be artificially created. For example, the image501can be artificially created by preparing a background image and superimposing a human face image at an appropriate position on the background image. If information as to which position of the background image the face image was superimposed is made metadata, a ground truth feature vector map corresponding to the image501can be generated without estimation of the face position. By the above-described method, training data can be created.

Next, the feature vector map extractor is trained by using the created training data. The feature vector map extractor is designed by a deep neural network called a Fully Convolutional Network, represented, for example, by U-Net and the like.

The feature vector map extractor is trained by using error back-propagation or the like so that the output obtained when the image of the training data is given to the feature vector map extractor approaches the ground truth feature vector map of the training data. The feature vector map extractor learning unit207stores the parameters of this feature vector map extractor in the storage device12as feature vector map extractor parameters209.

The above-described explanation assumes the use of a network derived from a CNN (Convolutional Neural Network) that serves as a feature vector map extractor, but a network derived from a ViT (Vision Transformer) can also be used. For example, by using the SimVIT block disclosed in the paper “Simvit: Exploring a simple vision transformer with sliding windows” by Li, Gang, et al., presented at the 2022 IEEE International Conference on Multimedia and Expo (ICME), it is possible to design a feature vector map extractor that outputs a feature map having a height of H/Kh, width of W/Kw, and depth of D.

Next, an example of the flow of the search process of the image processing apparatus100according to the present embodiment will be explained with reference toFIG.6andFIG.7.FIG.6is a flowchart showing the flow of the search process by the image processing apparatus in the present embodiment.FIG.7is a diagram showing an overall system for searching for a specific face. It should be noted that the operation of each step of the flowchart ofFIG.6is performed by a CPU that serves as a computer in a control device performing control by executing a computer program stored in a memory.

In step S601(image acquisition step), the image acquisition unit201acquires a query image301, which is an image that includes a subject to be searched, and an image to be searched304, which may include the search target. In the query image301, only one query face appears, and no other faces except for the query face are captured. The height of the image to be searched304is H and the width of the image to be searched304is W.

In step S602, the feature vector extraction unit202detects a query face302from the query image301that has been acquired by the image acquisition unit201, and extracts a feature vector303of the query face302. Here, step S602functions as a feature vector extraction step for extracting a first feature vector that represents features of a search target included in the query image.

Details of feature vector extraction will be explained. First, the feature vector extraction unit202generates a normalized query image by the same method that was explained in step S301. Then, the feature vector extraction unit202extracts the feature vector303of the query face302from the normalized query image by using the feature extractor parameters208that have been learned in advance in step S301. The feature vector303is a vector of a dimensionality D.

In step S603, the feature vector map extraction unit203extracts a feature vector map310from the image to be searched304that was acquired by the image acquisition unit201. The image to be searched304is an image in which the same person as the query face302may appear. Here, it is assumed that a face305is the face of the same person as the query face302. Here, step S603functions as a feature vector map extraction step for extracting a feature vector map that represents feature vectors of a subject at each position of the image to be searched.

The feature vector map extraction unit203extracts a feature vector map310from the image to be searched304by using the feature vector map extractor parameters209that have been learned in advance in step S302. The feature vector map310is a map having a height of H/Kh, a width of W/Kw, and a depth of D. The feature vector at the position (x/Kw, y/Kh) in this map represents the facial feature amount of a face in the vicinity of the position (x, y) of the image to be searched304.

In step S604, the heat map calculation unit204calculates a heat map311based on the feature vector303that was extracted by the feature vector extraction unit202and the feature vector map310that was extracted by the feature vector map extraction unit203. Here, step S604functions as a heat map calculation step for performing an arithmetic operation between the first feature vector and the feature vector map, and calculating a heat map that represents the likelihood that a search target is present.

The heat map311is a map with a height of H/Kh, a width of W/Kw, and a depth of 1. The value of the position (x, y) of the heat map311is determined by the similarity (for example, inner product) between the feature vector at the position (x, y) of the feature vector map310and the feature vector303. The calculation of the311heat map described above is equivalent to a 1×1 convolution operation. Therefore, it is possible to consider the feature vector303as the weight of the 1×1 convolution, and configure modules or circuits for a convolution operation so as to calculate the heat map311.

The heat map311can be interpreted as a map representing the likelihood that the query face302is present. That is, in a case in which the center of the query face302is at position (x1, y1) in the image to be searched304, the value of position (x1/Kw, y1/Kh) of the heat map311is expected to be a maximum value.

In addition, the value of the heat map311is expected to become smaller the farther away from the position (x1/Kw, y1/Kh). Furthermore, in a case in which there is no face in the vicinity of the position (x2, y2) of the image to be searched304, or in a case in which there is a face of a person different from the query face302, the value of the position (x2/Kw, y2/Kh) of the heat map311is expected to be a value close to zero.

In step S605, the face position estimation unit205estimates candidate positions in the image to be searched304in which the query face302may appear based on the heat map311that has been estimated by the heat map calculation unit204.

The estimation of a candidate position can be performed by selecting a position in the heat map311such that the value thereof is a maximum and is equal to or greater than a threshold value. That is, the candidate position in the image to be searched is determined based on the position where in the heatmap calculated by the heat map calculation unit204, the value thereof is a maximum and is equal to or greater than a threshold value.

In a case in which candidate positions are locally clustered, the number of candidate positions may be reduced by performing processing for merging candidate positions that are within a predetermined distance from each other. The candidate position (x3, y3) obtained on the heat map311correspond to the candidate position (x3/Kw, y3/Kh) in the image to be searched304.

A position312and a position313inFIG.7are examples of candidate positions in the heat map311. The candidate position312in the heat map311corresponds to the face305and the candidate position313in the image to be searched304corresponds the face306in the image to be searched304, indicating the possibility that the face305and the face306are the same person as the query face302.

In a case in which the number of candidate positions that have been estimated by the face position estimation unit205is 0, it is determined that the query face302has not been captured in the image to be searched304. In addition, in a case in which the number of candidate positions that have been estimated by the face position estimation unit205is one or more, it is determined that the query face302was captured in the image to be searched304. The face position estimation unit205may display an image to be searched on a display device via the output device14by generating a search image with a superimposed display, such as a circle or a rectangle, around estimated candidate positions.

As explained above, according to the image processing apparatus100of the First Embodiment, in a case in which a large number of small subjects appear in an image to be searched, it is possible to quickly estimate whether a certain subject appears, and the location of candidate positions in which a specific subject is likely to appear.

Second Embodiment

In a second embodiment, a case in which high-accuracy face recognition is performed in addition to the face search that was described in the First Embodiment will be explained. A description of similarities to the First Embodiment will be omitted, and only differences from the First Embodiment will be explained.

FIG.8is a functional block diagram showing an example of the functional configuration of an image processing apparatus according to the Second Embodiment. The image processing apparatus100includes a face position estimation unit801and a high-accuracy face authentication unit802. The face position estimation unit801estimates candidate positions in the image to be searched in which a query face is likely to appear, based on the heat map that has been calculated by the heat map calculation unit204. The candidate positions estimated by the face position estimation unit801are sent to the high-accuracy face authentication unit802.

For each face in the candidate positions estimated by the face position estimation unit801, the high-accuracy face authentication unit802functions as a determination unit that uses high-accuracy face authentication to determine with high accuracy whether the query face appears.

An example of the processing flow of the image processing apparatus100according to the present embodiment will be explained with reference toFIG.9.FIG.9is a flowchart showing the flow of a search process by the image processing apparatus according to the Second Embodiment. It should be noted that the operation of each step of the flowchart ofFIG.9is performed by a CPU that serves as a computer in a control device performing control by executing a computer program stored in a memory. Because step S601to step S605inFIG.9are the same process as that of step S601to step S605inFIG.6, an explanation thereof will be omitted.

In step S901, the high-accuracy face authentication unit802determines with high accuracy whether the query face appears by using high-accuracy face authentication for each face in a candidate position estimated by the face position estimation unit801. Hereinafter, a case in which highly accurate face authentication is performed with respect to a face in the candidate position312in the heat map311ofFIG.7will be explained.

A face image that is subject to highly accurate face authentication is acquired from the image to be searched304. When the position of the candidate position312in the heat map311is denoted as (x4, y4), the candidate position of the image to be searched304corresponding thereto is (x4/Kw, y4/Kh). A face is detected by applying a known face detection means to the vicinity (surroundings) of the position (x4/Kw, y4/Kh) of the image to be searched304.

Here, it is possible to detect faces of different sizes by performing face detection processing on each of a group of images in which an image to be searched304is resized to a slightly different size, and integrating the face detection results. After detecting a face, the generation of a normalized face image and the extraction of a feature vector using the feature extractor parameters208are performed in the same manner as that explained in step S602.

This feature vector can be expected to be the feature vector of the face305. The similarity between the extracted feature vector and the feature vector303of the query face302is calculated, and whether or not the face305is the same person as the query face302is determined with high accuracy according to whether or not the similarity is greater than a preset threshold value.

That is, in the Second Embodiment, at each candidate position in the image to be searched, a subject image is acquired from the image to be searched, and a second feature vector is extracted by applying the feature vector extraction means to the subject image. Then, an arithmetic operation is performed between the first feature vector and the second feature vector, and it is determined whether or not the search target is present at a candidate position in the image to be searched.

In contrast to a conventional face search system, in which face detection is first performed on an image to be searched and highly accurate face recognition processing is performed on each detected face, the face search system of the present embodiment performs highly accurate face recognition on only a small number of candidates narrowed down by the heat map. Therefore, the face search time can be significantly reduced in a case in which a large number of small subjects appear in the image to be searched.

It is also possible to change the emphasis on either accuracy or speed of face search by a setting. In the case of accuracy emphasis, the number of candidate positions can be increased by decreasing the threshold value for selecting candidate positions in step S605, or by increasing the number of parameters of the feature vector map extractor or feature extractor, but the processing time increases.

In the case of speed emphasis, the number of candidate positions can be decreased by increasing the threshold value for selecting candidate positions in step S605, or by decreasing the number of parameters of the feature vector map extractor or feature extractor, but the accuracy decreases. It may be possible to enable a user to select speed emphasis or accuracy emphasis by using a UI having, for example, a slide bar and the like.

As explained above, according to the image processing apparatus100of the Second Embodiment, in a case in which a large number of small subjects appear in an image to be searched, it is possible to quickly limit candidate positions in which a predetermined subject (query subject) may appear in a heat map. Then, it is possible to search for a specific subject at high speed and with high accuracy by performing highly accurate subject determination only for candidate positions.

Third Embodiment

Similar to the Second Embodiment, in a Third Embodiment, in addition to the face search that was described in the First Embodiment, a case in which high-accuracy face recognition is performed will be explained. A description of similarities to the First Embodiment and Second Embodiment will be omitted, and only differences from the First Embodiment and Second Embodiment will be explained.

FIG.10is a functional block diagram showing an example of a functional configuration of the image processing apparatus according to the Third Embodiment. The image processing apparatus100includes a face position estimation unit1001, a high-resolution image acquisition unit1002, and a high-accuracy face authentication unit1003.

The face position estimation unit1001estimates candidate positions in the image to be searched in which a query face is likely to appear, based on the heat map that has been calculated by the heat map calculation unit204. The candidate positions estimated by the face position estimation unit1001are sent to the high-resolution image acquisition unit1002.

The high-resolution image acquisition unit1002acquires a high-resolution image of a face corresponding to a candidate position on the heat map estimated by the face position estimation unit1001by controlling a camera such as a PTZ (pan, tilt, zoom) camera.

Here, at each candidate position in the image to be searched, the high-resolution image acquisition unit1002functions as a high-resolution image acquisition unit for acquiring a higher-resolution subject image. The high-resolution image acquired by the high-resolution image acquisition unit1002is sent to the high-accuracy face authentication unit1003.

The high-accuracy face authentication unit1003functions as a determination unit that uses high-accuracy face authentication to determine with high accuracy whether a query face appears in the high-resolution image that has been acquired by the high-resolution image acquisition unit1002.

An example of the processing flow of the image processing apparatus100according to the present embodiment will be explained with reference toFIG.11.FIG.11is a flowchart showing a flow of a search process by the image processing apparatus according to the Third Embodiment. It should be noted that the operation of each step of the flowchart ofFIG.11is performed by a CPU that serves as a computer in a control device performing control by executing a computer program stored in a memory. Because step S601to step S605inFIG.11are the same process as that of step S601to step S605inFIG.6, an explanation thereof will be omitted.

In step S1101, the high-resolution image acquisition unit1002acquires a high-resolution image of a face corresponding to a candidate position on the heat map estimated by the face position estimation unit1001by controlling a camera such as a PTZ camera. Hereinafter, a case in which highly accurate face authentication is performed with respect to a face in the candidate position312in the heat map311ofFIG.7will be explained.

A face image that is subject to highly accurate face authentication is acquired by controlling a camera such as a PTZ camera. The PTZ camera is calibrated in advance so as to be able to acquire a zoomed-in image of an arbitrary position in the image to be searched.

When the position of the candidate position312in the heat map311is denoted as (x4, y4), the position of the image to be searched304corresponding thereto is (x4/Kw, y4/Kh). A high-resolution image corresponding to the position (x4/Kw, y4/Kh) of the image to be searched304is acquired by driving the PTZ camera to, for example, point the PTZ camera at the position (x4/Kw, y4/Kh) and zoom in. Therefore, in this high-resolution image, the face305will appear more clearly than before driving the PTZ camera.

The high-accuracy face authentication unit1003uses high-accuracy face authentication to determine with high accuracy whether a query face appears in the high-resolution image acquired by the high-resolution image acquisition unit1002. Hereinafter, an example thereof will be explained.

The high-accuracy face authentication unit1003receives the high-resolution image acquired by the high-resolution image acquisition unit1002in step S1101. Next, after detecting a face by applying a known face detection technique to the high-resolution image, the generation of a normalized face image and the extraction of a feature vector using the feature extractor parameters208are performed in the same manner as that explained in step S602.

This feature vector can be expected to be the feature vector of the face305. The similarity between the extracted feature vector and the feature vector303of the query face302is calculated, and whether or not the face305is the same person as the query face302is determined according to whether or not the similarity is greater than a preset threshold value.

Thus, in the Third Embodiment, at each candidate position in an image to be searched, a high-resolution image is acquired by using the high-resolution image acquisition unit1002. Then, a second feature vector is extracted by applying the feature vector extraction unit202to the high-resolution image, an arithmetic operation is performed between the first feature vector and the second feature vector, and it is determined whether or not the search target is present at the candidate position in the image to be searched.

Similar to the Second Embodiment, the Third Embodiment can significantly reduce face search time in a case in which a large number of small subjects are captured in the image to be searched. Furthermore, to perform highly accurate face recognition based on a high-resolution image, it becomes possible to determine whether a specific face is present with higher accuracy as compared to the Second Embodiment. Moreover, according to the image processing apparatus in each embodiment, it is possible to search for a predetermined subject at high speed in a case in which a large number of small subjects are present in an image to be searched and the like.

OTHER EMBODIMENTS

Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s).

The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium.

The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No. 2022-179814, filed on Nov. 9, 2022, which is hereby incorporated by reference herein in its entirety.