Patent Description:
The introduction of non-face-to-face meetings and/or telecommuting has been boosted by COVID-<NUM>. As a result, systems for providing a work environment for telecommuting equivalent to office work have been actively established. The expansion of the scope of allowance of access authority to a system for more efficient telecommuting environments increases the accessibility to various work data and the risk of leak of confidential information as well. Examples of systems deploying face recognition are for example known from "<NPL> or from Japanese application <CIT>.

In one general aspect, a method with access authority management according to claim <NUM> is provided.

The determining of whether the user corresponds to the multiple users or the single user may include determining whether the user corresponds to the multiple users or the single user based on any one or any combination of any two or more of a number of the region of the at least one portion of the body, a color of the region of the at least one portion of the body, a position of disposition of the region of the at least one portion of the body compared to a position of the face region, an angle of disposition, and a size ratio of the region of the at least one portion of the body to the face region.

The performing may include: extracting a feature vector of the face region from the input image; and performing the verification for the user based on a result of comparing a similarity score between the feature vector of the face region and an enrolled feature vector stored in an enrollment database to a verification threshold.

The method may include adjusting the verification threshold according to a security level corresponding to the system, wherein the performing may include performing the verification for the user according to whether the similarity score is greater than the adjusted verification threshold.

The determining of whether the input image is the real image or the spoofed image may include: extracting an output feature vector of a neural network for detecting whether the face region is spoofed from the input image; calculating a score based on the output feature vector; and determining whether the input image is the real image or the spoofed image according to whether the score falls within a threshold range for determining whether the input image is spoofed.

The method may include adjusting the threshold range according to a security level corresponding to the system, wherein the determining of whether the input image is the real image or the spoofed image may include determining whether the input image is the real image or the spoofed image according to whether the score falls within the adjusted threshold range.

The method includes: determining whether a preset type of capturing device is detected in the input image; and blocking an access authority of the user to the system in response to the determination that the preset type of capturing device is detected.

The method may include: determining whether an abnormal situation is detected in a second input image received after the access authority is allowed, the abnormal situation including any one or any combination of any two or more of another user other than the authorized user, the preset type of capturing device, and a capturing action using the capturing device; and blocking an access authority to the system in response to the determination that an abnormal situation is detected.

The determining of whether the abnormal situation is detected may include determining whether the other user is detected based on whether a plurality of face regions are detected in the second input image.

The determining of whether the abnormal situation is detected may include determining whether an object corresponding to the preset type of capturing device is detected in the second input image.

The determining of whether the abnormal situation is detected may include determining whether the capturing action, being of the user and corresponding to the preset type of capturing device, is detected in the second input image.

The determining of whether the capturing action of the user is detected may include determining whether the capturing action of the user is detected based on whether any one or any combination of any two or more of a pose of the user facing a screen provided through the system, a gesture of the user, a position of the capturing device corresponding to the screen, and consecutive motions of the user using the capturing device in the second input image corresponds to a motion predetermined as a capturing action.

The method may include guiding the user to transmit an image of a space outside a field of view corresponding to the input image by rotating a capturing device used to capture the input image and capturing the space, according to a security level corresponding to the system.

The method may include verifying whether the input image is captured by a capturing device with a polarizing film attached to narrow a field of view.

The verifying may include verifying, in response to the user wearing glasses, whether the input image is captured by a capturing device with a polarizing film attached to narrow a field of view, based on either one or both of a pattern and a color reflected on a lens of the glasses corresponding to the input image.

The receiving of the input image may include either one or both of: receiving the input image according to a preset time interval; and receiving the input image according to a random time interval.

The input image may include an image obtained by capturing a working situation of the user who is telecommuting.

In another general aspect, one or more embodiments include a non-transitory computer-readable storage medium according to claim <NUM>.

In another general aspect, an apparatus with access authority management according to claim <NUM> is provided.

In another general aspect, a method with access authority management according to claim <NUM> is provided.

Likewise, each of expressions, for example, "between" and "immediately between" and "adjacent to" and "immediately adjacent to," should also be respectively construed in the same way.

The use of the term "may" herein with respect to an example or embodiment (e.g., as to what an example or embodiment may include or implement) means that at least one example or embodiment exists where such a feature is included or implemented, while all examples are not limited thereto.

Unless otherwise defined, all terms used herein including technical or scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and based on an understanding of the disclosure of the present application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of the present application, and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Hereinafter, examples will be described in detail with reference to the accompanying drawings. When describing the examples with reference to the accompanying drawings, like reference numerals refer to like components and a repeated description related thereto will be omitted.

<FIG> illustrates an example of an overview of an apparatus for managing an access authority. Operations shown in <FIG> are not necessarily to be sequentially arranged as illustrated. According to examples, the operations may be performed in different orders, and at least two of the operations may be performed in parallel.

Referring to <FIG>, a process of performing screen unlock or maintaining screen lock by an apparatus <NUM> for managing an access authority (hereinafter, the "management apparatus" <NUM>) through a determination of whether a user corresponding to an input image <NUM> is a user having an access authority to a system is illustrated.

The management apparatus <NUM> may be, for example, a telecommuting management system itself that manages telecommuting situations of users, or a management apparatus separate from the telecommuting management system.

The management apparatus <NUM> may be intended to enhance the security of the telecommuting management system, and may allow the access of a user whose access to the telecommuting management system is authorized ("authorized user") and block the access of user(s) whose access is not authorized ("unauthorized user(s)"). The management apparatus <NUM> may provide an access authority to a system (e.g., the telecommuting management system) to a user by determining whether the user is an authorized user or an unauthorized user by utilizing unique biometric information (e.g., face) of the user.

Also, the management apparatus <NUM> may block an access to the system by detecting an unauthorized user, a capturing device, and/or a capturing action using a capturing device from the input image <NUM>. For example, after an authorized user accesses the telecommuting management system, the management apparatus <NUM> may detect from the received input image <NUM> that the authorized user is with another user (an unauthorized user) in the same space or that the authorized user performs a capturing action using a capturing device. In this case, the management apparatus <NUM> may determine that the detection result (e.g., the detection of an unauthorized user, the detection of a capturing device, and/or the detection of a capturing action) corresponds to an abnormal situation of information leakage attempt and thus, may block an access authority of the authorized user to the telecommuting management system.

The management apparatus <NUM> may, for example, perform object detection process <NUM>, face recognition process <NUM>, and template update process <NUM>.

For example, the management apparatus <NUM> may receive an input image <NUM>. The input image <NUM> may include, for example, at least one portion of a body of an authorized user A and/or another object.

In object detection process <NUM>, the management apparatus <NUM> may detect the object from the input image <NUM>. Here, the object may correspond to, for example, a body part such as a face, a hand or an arm of a person, and may also include a capturing device such as a camera and/or a user device such as a smartphone or a laptop computer. However, examples are not necessarily limited thereto.

The management apparatus <NUM> may detect a capturing device such as a camera or a smartphone, in addition to at least one portion of a body of the authorized user A (e.g., a face of the user A) from the input image <NUM>. The management apparatus <NUM> may detect the capturing device according to, for example, the shape of the capturing device and/or how the capturing device is used included in the input image <NUM>. However, examples are not necessarily limited thereto.

When an object such as a capturing device is detected from the input image <NUM> in object detection process <NUM>, the management apparatus <NUM> may block an access authority to the system granted to the user A. The management apparatus <NUM> may perform screen lock <NUM> to lock the screen provided to the user A as the access authority of the user A is blocked.

When the capturing device is not detected in object detection process <NUM>, the management apparatus <NUM> may verify through face recognition process <NUM> whether at least two faces are detected in the input image <NUM> and/or whether at least one portion of the body of the user detected in the input image <NUM> corresponds to a face region of the authorized user (the user A).

In face recognition process <NUM>, the management apparatus <NUM> may determine whether the number of face regions included in the input image <NUM> is greater than or equal to "<NUM>", that is, whether at least two faces are detected. In response to the determination that the number of face regions included in the input image <NUM> is "<NUM>", the management apparatus <NUM> may verify whether the face included in the input image <NUM> is the face of the authorized user A. According to an example, in face recognition process <NUM>, the management apparatus <NUM> may determine whether the input image <NUM> is a real image or a spoofed image.

For example, when at least two faces are detected in the input image <NUM>, the management apparatus <NUM> may determine that a verification for the face region of the user A included in the detected region of at least one portion of the body of the user is failed and/or may determine that the face of the user A is not a real image but a spoofed image. When the management apparatus <NUM> determines that the verification is failed and/or the face of the user A is the spoofed image, the management apparatus <NUM> may block the access authority for the system granted to the user A (e.g., the management apparatus <NUM> may perform screen lock <NUM> to lock the screen provided to the user A).

When the verification for the input image <NUM> is successful through face recognition process <NUM> and it is determined that the input image <NUM> is a real image, the management apparatus <NUM> may perform template update process <NUM>.

Object detection process <NUM> and face recognition process <NUM> may be performed in various manners, non-limiting examples of which will be described in more detail with reference to <FIG>.

In template update process <NUM>, the management apparatus <NUM> may update template(s) stored in an enrollment database (DB) based on the input image <NUM>. Templates may correspond to enrolled images of the user A. In template update process <NUM>, the management apparatus <NUM> may update the templates by newly enrolling the input image <NUM> as an enrolled image in the enrollment DB or replacing an enrolled image stored in the enrollment DB with the input image <NUM>. A non-limiting example of the process of enrolling an input image in the enrollment DB by the management apparatus <NUM> will be described in more detail with reference to <FIG>.

When the update of the templates is completed through template update process <NUM>, the management apparatus <NUM> may allow an access authority to the system to the user A (e.g., the management apparatus <NUM> may perform screen unlock <NUM> to unlock the screen provided to the user A). According to an example, when the verification for the input image <NUM> is successful through face recognition process <NUM> and it is determined that the input image <NUM> is a real image, the management apparatus <NUM> may allow an access authority to the system to the user A in response to the completion of face recognition process <NUM>, prior to updating the templates.

<FIG> illustrates an example of a process of enrolling an enrolled image for user verification by a management apparatus. Referring to <FIG>, a process of enrolling an input image <NUM> of a user as an enrolled image in an enrollment DB by a management apparatus is illustrated.

For example, when an input image <NUM> captured by a webcam or a camera is received, the management apparatus may detect objects from the input image <NUM>, in operation <NUM>. The management apparatus may detect, for example, whether a region of at least one portion of a body of the user, a capturing device, and/or a face is included, from the input image <NUM>. The input image <NUM> may be, for example, a two-dimensional image including a face region of the user and/or a region of at least one portion of the body of the user. However, examples are not necessarily limited thereto.

The management apparatus may detect the region of at least one portion of the body of the user, the capturing device, and/or the face using a neural network. The neural network may be, for example, pre-trained to detect a region of at least one portion of a user, a capturing device, and/or a face. The neural network may determine the number and positions of objects included in the input image <NUM> through bounding boxes corresponding to classes of the respective objects and transmit the determined number and positions of the objects to the management apparatus.

The management apparatus may determine whether a preset type of capturing device is among the objects detected in operation <NUM>. In response to the determination that a preset type of capturing device is detected in operation <NUM>, the management apparatus may exclude the input image <NUM> from enrollment in operation <NUM>.

In response to the determination that a preset type of capturing device is not detected in operation <NUM>, the management apparatus may determine whether the detected object is a single user, in operation <NUM>. When it is determined in operation <NUM> that the detected object is not a single user, the management apparatus may exclude the input image <NUM> from enrollment in operation <NUM>.

When it is determined in operation <NUM> that the detected object is a single user, the management apparatus may determine a liveness of the input image <NUM> in operation <NUM>. "Liveness" may correspond to whether an image (the input image <NUM>) is a real image or a spoofed image. Here, "spoofing" refers to fake biometric information other than live biometric information, and may be construed as including, for example, duplication, forgery, and falsification of biometric information (e.g., a face image).

When it is determined that the input image <NUM> is not an actually captured image but a duplicated or forged photo or movie, or a mask with the face of the user drawn, the management apparatus may determine that the input image <NUM> is a spoofed image. When it is determined in operation <NUM> that the input image <NUM> is a spoofed image, the management apparatus may exclude the input image <NUM> from enrollment in operation <NUM>.

When it is determined in operation <NUM> that the input image <NUM> is a real image, the management apparatus may perform a user verification by the face region of the input image <NUM> in operation <NUM>. When the user verification is failed in operation <NUM>, the management apparatus may exclude the input image <NUM> from enrollment in operation <NUM>.

When the user verification is successful in operation <NUM>, the management apparatus may enroll the input image <NUM> in an enrollment DB in operation <NUM>. An image enrolled in the enrollment DB may be utilized in a future user verification process and compared to an input image. The enrollment DB may store, for example, an enrolled image itself, and/or may store features of a face region included in the enrolled image. The features of the face region may be stored in the form of a feature vector or a feature map. However, examples are not necessarily limited thereto. A non-limiting example of the process of performing a user verification by the management apparatus will be described in more detail with reference to <FIG>.

<FIG> illustrates an example of a method of managing an access authority. In the following example, operations may be performed sequentially, but not necessarily performed sequentially. For example, the operations may be performed in different orders, and at least two of the operations may be performed in parallel.

Referring to <FIG>, a process of allowing an access authority to a system to a user by a management apparatus through operations <NUM> to <NUM> is illustrated.

In operation <NUM>, the management apparatus receives an input image including a region of at least one portion of a body of a user. The input image may be, for example, an image obtained by capturing a working situation of the user who is telecommuting. The input image may be a two-dimensional image or a three-dimensional image.

In operation <NUM>, the management apparatus may receive the input image according to a preset time interval, or may receive the input image according to a random time interval. According to an embodiment, the management apparatus may receive the input image at a time interval in which a predetermined time interval and a random time interval are mixed.

In operation <NUM>, the management apparatus determines whether the user is multiple users or a single user using the region of the at least one portion of the body included in the input image received in operation <NUM>. For example, the management apparatus may determine whether the user is multiple users or a single user based on at least one of a number of the region of the at least one portion of the body, a color of the region of the at least one portion of the body, a position of disposition of the region of the at least one portion of the body compared to a position of a face region, an angle of disposition, and/or a size ratio of the region of the at least one portion of the body to the face region.

For example, when the number of regions of at least one portion of the body (e.g., arms or hands) included in the input image exceeds the normal number of body parts (e.g., "<NUM>" exceeding a preset normal number "<NUM>" for arms or hands), the management apparatus may determine that the user included in the input image is multiple users. Alternatively or additionally, when a color difference between body parts of the user exceeds a predetermined standard, such as when the color of the arm of the user included in the input image is dark brown whereas the color of the face of the user is a lighter hue, for example, yellow or white, the management apparatus may determine that the user included in the input image is multiple users. In addition, when the position of disposition of the region of the at least one portion of the body (e.g., the arm or neck) compared to the position of the face region included in the input image, the angle of disposition, and/or the size ratio of the region of the at least one portion of the body to the face region (e.g., a thickness ratio of an arm or neck region compared to the size of the face region) exceed a position of disposition, an angle of disposition, and/or a size ratio of typical users, the management apparatus may determine that the user included in the input image is multiple users.

When it is determined in operation <NUM> that the user included in the input image is multiple users, the management apparatus may block an access authority of the user to the system.

In operation <NUM>, the management apparatus performs a verification for the user by the face region in the input image in response to the determination of operation <NUM> (e.g., the determination that the user is a single user). In operation <NUM>, the management apparatus may extract a feature vector of the face region from the input image. The management apparatus may perform the verification for the user based on a result of comparing the feature vector of the face region to an enrolled feature vector stored in an enrollment DB. For example, the management apparatus may perform the verification for the user based on a result of comparing a similarity score between the feature vector of the face region and the enrolled feature vector stored in the enrollment DB to a verification threshold. In this case, the management apparatus may adjust the verification threshold to be high or low according to a security level corresponding to the system.

For example, the security level corresponding to the system may range from the lowest level "<NUM>" to the highest level "<NUM>". When the security level corresponding to the system is set to "Level <NUM>" which is higher than a reference level (e.g., "Level <NUM>"), the management apparatus may adjust the verification threshold to be higher than a default value (or an initial verification threshold). Conversely, when the security level corresponding to the system is set to "Level <NUM>" which is lower than the reference level, the management apparatus may adjust the verification threshold to be lower than the default value. The management apparatus may perform the verification for the user according to whether the similarity score is greater than the adjusted verification threshold. In other words, when the similarity score between the feature vector of the face region and the enrolled feature vector stored in the enrollment DB is greater than or equal to the verification threshold adjusted according to the security level of the system, the management apparatus may determine that the user verification is successful. When the similarity score is less than the verification threshold adjusted according to the security level of the system, the management apparatus may determine that the user verification is failed. When the verification performed in operation <NUM> is failed, the management apparatus may block the access authority of the user to the system.

In operation <NUM>, the management apparatus determines whether the input image is a real image or a spoofed image based on whether the verification performed in operation <NUM> is successful. In operation <NUM>, the management apparatus may extract an output feature vector of a neural network by applying the input image to the neural network configured to detect whether a face region is spoofed from the input image. The management apparatus may calculate a score based on the output feature vector. The management apparatus may determine whether the input image is a real image or a spoofed image according to whether the calculated score falls within a threshold range for determining whether the input image is spoofed. When it is determined in operation <NUM> that the input image is a spoofed image, the management apparatus may block the access authority of the user to the system.

According to an example, the management apparatus may adjust the threshold range according to the security level corresponding to the system (e.g., when the security level corresponding to the system is adjusted). In this case, the management apparatus may determine whether the input image is a real image or a spoofed image according to whether the score falls within the adjusted threshold range.

The management apparatus may adjust the level of determination difficulty by changing the verification threshold for determining whether the verification is successful in operation <NUM> and/or the threshold range for determining a liveness in operation <NUM> according to the security level corresponding to the system, thereby more flexibly regulating the trade-off between usability and security. Alternatively or additionally, according to an example, the management apparatus may guide (e.g., instruct) the user to transmit an image (e.g., a panoramic image (video) or a single image) of a space outside a field of view corresponding to the input image by rotating a capturing device used to capture the input image and capturing the space, according to the security level corresponding to the system. For example, the management apparatus may cause the user to transmit an image of a space outside a field of view of a webcam used to capture the input image by capturing the space using a separate capturing device. The management apparatus may enhance the security by checking whether there is another user through the image captured by the separate capturing device.

In operation <NUM>, the management apparatus may allow an access authority to the system to the user, in response to the determination of operation <NUM> (e.g., the determination that the input image is a real image). According to an example, the management apparatus may determine whether a preset type of capturing device (e.g., a camera, a cell phone, a pen-type capturing device, and/or a pin-type capturing device) is detected in the input image, and block an access authority of the user to the system in response to the determination that a capturing device is detected.

According to an example, the management apparatus may additionally verify whether the input image is captured by a capturing device with a polarizing film attached to narrow a field of view. The management apparatus may verify, in response to the user wearing glasses, whether the input image is captured by a capturing device with a polarizing film attached to narrow a field of view, based on at least one of a pattern and a color reflected on a lens of the glasses corresponding to the input image. For example, the management apparatus may verify whether the input image is captured by a capturing device with a polarizing film attached to narrow a field of view, according to a degree of deformation of the pattern or color reflected on the lens of the glasses. However, examples are not necessarily limited thereto.

<FIG> illustrates an example of a detection process performed for allowing an access authority to a system by a management apparatus. Referring to <FIG>, a process of detecting a capturing device and multiple users from an input image <NUM> by a management apparatus is illustrated.

In operation <NUM>, a management apparatus may detect an object from the input image <NUM>. For example, the management apparatus may detect, from the input image <NUM>, whether a region of at least one portion of a body of a user, a capturing device, and/or a face is included.

In operation <NUM>, the management apparatus may detect a region of at least one portion of a body of a user, a capturing device, and/or a face using a neural network pre-trained to detect a region of at least one portion of a body of a user, a capturing device, and/or a face. In this case, the neural network may determine the number and positions of objects included in the input image <NUM> through bounding boxes corresponding to classes of the respective objects (e.g., a region of at least one portion of a body of a user, a capturing device, and/or the like) and transmit the determined number and positions of the objects to the management apparatus.

In operation <NUM>, the management apparatus may determine whether a preset type of capturing device is among the objects detected in operation <NUM>. In response to the determination that a preset type of capturing device is detected in operation <NUM>, the management apparatus may block an access authority of the user to a system by locking a screen provided to the user, in operation <NUM>.

In response to the determination that a preset type of capturing device is not detected in operation <NUM>, the management apparatus may determine a liveness of the input image <NUM>, that is, whether the input image <NUM> is a real image or a spoofed image, in operation <NUM>.

In operation <NUM>, for example, when the input image <NUM> is not an actually captured image but a duplicated or forged photo or movie, or a mask with the face of the user drawn, the management apparatus may determine that the input image <NUM> is a spoofed image. When it is determined in operation <NUM> that the input image <NUM> is a spoofed image, the management apparatus may block the access authority of the user to the system by locking the screen provided to the user, in operation <NUM>.

When it is determined in operation <NUM> that the input image <NUM> is a real image, the management apparatus may determine whether the input image <NUM> includes multiple users, in operation <NUM>. In operation <NUM>, as described above, the management apparatus may determine whether the user included in the input image <NUM> is multiple users or a single user based on at least one of a number of the region of the at least one portion of the body included in the input image <NUM>, a color of the region of the at least one portion of the body included in the input image <NUM>, positions of dispositions of regions of other portions of the body compared to a position of a face region in the input image <NUM>, angles of dispositions, and/or a size ratio of the region of the at least one portion of the body to the face region.

When it is determined in operation <NUM> that the input image <NUM> includes multiple users, the management apparatus may block the access authority of the user to the system by locking the screen provided to the user, in operation <NUM>.

When it is determined in operation <NUM> that the input image <NUM> includes a single user, the management apparatus may extract a face feature of the user from the face region included in the input image <NUM>, in operation <NUM>. The management apparatus may extract the face feature of the user, for example, in the form of a feature vector or a feature map. However, examples are not necessarily limited thereto.

In operation <NUM>, the management apparatus may compare the feature extracted in operation <NUM> with an enrolled feature stored in an enrollment DB <NUM>. In this case, the enrolled feature stored in the enrollment DB <NUM> may correspond to a feature vector or a feature map of an enrolled image. However, examples are not necessarily limited thereto.

When it is determined that a comparison result in operation <NUM> is a match, the management apparatus may allow the access authority to the system to the user by unlocking the screen, in operation <NUM>. Here, "a comparison result is a match" may be construed as including both a case in which the feature extracted in operation <NUM> is completely the same as the enrolled feature stored in the enrollment DB <NUM> and a case in which the features have a similarity within the margin of error acceptable to acknowledge they are the same.

When it is determined that the comparison result in operation <NUM> is not a match, the management apparatus may block the access authority to the system by locking the screen provided to the user, in operation <NUM>.

In operation <NUM>, the management apparatus receives an input image obtained by capturing at least one portion of a body of a user.

In operation <NUM>, the management apparatus determines whether the user is multiple users or a single user using the at least one portion of the body included in the input image received in operation <NUM>. When it is determined in operation <NUM> that the user is multiple users, the management apparatus may block an access authority of the user to the system.

In operation <NUM>, the management apparatus determines whether the input image is a real image or a spoofed image using a face region in the input image, in response to the determination of operation <NUM> (e.g., the determination that the user is a single user). When it is determined in operation <NUM> that the input image is a spoofed image, the management apparatus may block the access authority of the user to the system.

In operation <NUM>, the management apparatus performs a verification for the user by the face region, in response to the determination of operation <NUM> (e.g., the determination that the input image is a real image). When the verification for the user is failed in operation <NUM>, the management apparatus may block the access authority of the user to the system.

In operation <NUM>, the management apparatus allows the access authority to the system to the user based on whether the verification performed in operation <NUM> is successful.

In operation <NUM>, similar to operation <NUM> described above, the management apparatus may detect a region of at least one portion of a body of a user, a capturing device, and/or a face using a neural network pre-trained to detect a region of at least one portion of a body of a user, a capturing device, and/or a face.

In operation <NUM>, the management apparatus determines whether a preset type of capturing device is among the objects detected in operation <NUM>. In response to the determination that a preset type of capturing device is detected in operation <NUM>, the management apparatus blocks an access authority of the user to a system by locking a screen provided to the user, in operation <NUM>.

In response to the determination that a preset type of capturing device is not detected in operation <NUM>, the management apparatus may extract a face feature of the user from a face region included in the input image, in operation <NUM>. In this case, the management apparatus may extract the face feature of the user in the form of a feature vector or a feature map. However, examples are not necessarily limited thereto.

In operation <NUM>, the management apparatus may determine a liveness of the input image <NUM>, that is, whether the input image <NUM> is a real image or a spoofed image, using the face feature of the user extracted in operation <NUM>. When it is determined in operation <NUM> that the input image <NUM> is a spoofed image, the management apparatus may block the access authority of the user to the system by locking the screen provided to the user, in operation <NUM>.

When it is determined in operation <NUM> that the input image <NUM> is a real image, the management apparatus may determine whether the input image <NUM> includes multiple users, in operation <NUM>. In operation <NUM>, as described above through operation <NUM> of <FIG>, the management apparatus may determine whether the user included in the input image <NUM> is multiple users or a single user based on at least one of a number of the region of the at least one portion of the body included in the input image <NUM>, a color of the region of the at least one portion of the body included in the input image <NUM>, positions of dispositions of regions of other portions of the body compared to a position of a face region in the input image <NUM>, angles of dispositions, and/or a size ratio of the region of the at least one portion of the body to the face region.

When it is determined in operation <NUM> that the input image <NUM> includes a single user, the management apparatus may compare the feature extracted in operation <NUM> with an enrolled feature stored in an enrollment DB <NUM>, in operation <NUM>. In this case, the enrolled feature stored in the enrollment DB <NUM> may correspond to a feature vector or a feature map of an enrolled image. However, examples are not necessarily limited thereto.

<FIG> illustrates an example of a process of performing a verification for a user by a management apparatus. Referring to <FIG>, a process of verifying whether a user B included in an input image <NUM> has an access authority to a system by matching the input image <NUM> with enrolled images <NUM> stored in an enrollment DB <NUM> by a management apparatus is illustrated.

For example, when the input image <NUM> including a face of the user B is received, the management apparatus may perform object detection on the input image <NUM>, in operation <NUM>. In operation <NUM>, the management apparatus may check whether object(s) detected in operation <NUM> correspond to predetermined conditions (e.g., whether the object(s) correspond to multiple users, correspond to a capturing device, and/or include a face region). When it is checked in operation <NUM> that the detected object(s) correspond to any one of the predetermined conditions, the management apparatus may skip a face verification on the input image <NUM>, and block the access authority of the user to the system by immediately lock the screen in operation <NUM>.

When it is checked in operation <NUM> that the detected object(s)do not correspond to the predetermined conditions, the management apparatus may perform a face verification for the detected object, in operation <NUM>.

In operation <NUM>, the management apparatus may compare the input image <NUM> with the enrolled images <NUM> stored in the enrollment DB <NUM> for the face verification on the input image <NUM>. In this case, the enrolled images <NUM> stored in the enrollment DB <NUM> may correspond to images of an authorized user A <NUM>. Similar to operations <NUM> and <NUM>, the enrolled images <NUM> may be enrolled in the enrollment DB <NUM> when it is checked that the images of the user A <NUM> do not correspond to the predetermined conditions, as a result of performing the process of objection detection <NUM> and check <NUM> on the images of the user A <NUM>.

When it is determined that the input image <NUM> matches the enrolled images <NUM> according to a comparison result of operation <NUM>, the management apparatus may allow the access authority of the user to the system by unlocking the screen, in operation <NUM>.

When it is determined that the input image <NUM> does not match the enrolled images <NUM> according to the comparison result of operation <NUM>, the management apparatus may block the access authority of the user to the system by locking the screen, in operation <NUM>.

In the example of <FIG>, since the input image <NUM> does not match the enrolled images <NUM>, the management apparatus may block the access authority of the user B to the system through screen lock.

<FIG> illustrates an example of a process of detecting a user by a management apparatus. Referring to <FIG>, a process of performing object and face detection on an input image <NUM> by a management apparatus prior to performing a verification <NUM> for a user corresponding to the input image <NUM> is illustrated.

When the input image <NUM> is received, the management apparatus may detect an object using an object detection model <NUM>. The object detection model <NUM> may detect an object by, for example, separate neural networks trained to detect multiple users, cameras, and cell phones, respectively. Alternatively or additionally, the object detection model <NUM> may detect an object through a single neural network trained to classify classes of objects detected by bounding boxes into multiple users, cameras, and cell phones.

More specifically, in operation <NUM>, the management apparatus may check whether multiple users are detected in the input image <NUM> using the object detection model <NUM>. When multiple users are detected in operation <NUM>, the management apparatus may determine that the process is failed in operation <NUM>, and lock a screen in operation <NUM>.

When multiple users are not detected in operation <NUM>, the management apparatus checks whether a camera is detected in the input image <NUM> using the object detection model <NUM>, in operation <NUM>. When a camera is detected in operation <NUM>, the management apparatus may determine that the process is failed in operation <NUM>, and lock the screen in operation <NUM>.

When a camera is not detected in operation <NUM>, the management apparatus may check whether a cell phone is detected in the input image <NUM> using the object detection model <NUM>, in operation <NUM>. When a cell phone is detected in operation <NUM>, the management apparatus may determine that the process is failed in operation <NUM>, and lock the screen in operation <NUM>.

When a cell phone is not detected in operation <NUM>, the management apparatus may determine that the process is successful in operation <NUM>, and perform face detection in operation <NUM>. In operation <NUM>, the management apparatus may detect a face region in the input image <NUM>.

In operation <NUM>, the management apparatus may determine whether the number of face regions detected in operation <NUM> is "<NUM>". When it is determined in operation <NUM> that the number of face regions is "<NUM>", the management apparatus may perform a verification for the user corresponding to the input image <NUM>, in operation <NUM>.

Conversely, when it is determined in operation <NUM> that the number of face regions is not "<NUM>", the management apparatus may determine whether the number of face regions is "<NUM>", in operation <NUM>. When it is determined in operation <NUM> that the number of face regions is "<NUM>", the management apparatus may not perform a verification for the user since a face region is not detected. Thus, the management apparatus may determine that the process is failed in operation <NUM>, and lock the screen in operation <NUM>. In addition, when it is determined in operation <NUM> that the number of face regions is not "<NUM>", it means that the number of face regions is greater than or equal to "<NUM>". Thus, the management apparatus may determine that the process is failed in operation <NUM>, and lock the screen in operation <NUM>.

<FIG> illustrates an example of a process of detecting a capturing device by a management apparatus. Referring to <FIG>, a process of detecting various types of capturing device(s) by a management apparatus prior to performing a face recognition from input image(s) <NUM> is illustrated.

When an input image <NUM> is received, the management apparatus may detect an object using an object detection model <NUM>. The object detection model <NUM> may detect an object by, for example, separate neural networks trained to detect multiple users, cameras, and cell phones, respectively. Alternatively or additionally, the object detection model <NUM> may detect an object through a single neural network trained to classify classes of objects detected by bounding boxes into multiple users, cameras, and cell phones.

When multiple users are not detected in operation <NUM>, the management apparatus checks whether a camera is detected in the input image <NUM> using the object detection model <NUM>, in operation <NUM>. In this case, the object detection model <NUM> may be trained to detect a typical type of camera with a protruding camera lens as well as a modified type of camera with a hidden camera lens, for example, a pen-type camera, a pin-type camera, and/or a bag-type camera.

Further, in operation <NUM>, the management apparatus may check or determine whether a capturing action of a user corresponding to a camera is detected in addition to a camera in the input image <NUM>. The management apparatus may determine whether a capturing action of the user is detected based on whether at least one of a pose of the user, a gesture of the user, a position of the capturing device, or consecutive motions of the user corresponding to the camera corresponds to a motion suspected as (e.g., predetermined to be) a capturing action.

The management apparatus may determine that a pose of the user facing a screen provided through a system (e.g., a pose of holding a camera in a hand, or a pose of pressing a shutter), a gesture of the user (e.g., a gesture of pressing a shutter or button of a camera), a position of a capturing device corresponding to the screen (e.g., a position of a capturing device facing the screen in the front), and/or consecutive motions of the user using a capturing device (e.g., consecutive motions of capturing the screen at a distance close to the screen and then, changing a page displayed on the screen to the next page, and capturing the screen displaying the next page again) correspond to a motion suspected as a capturing action. However, examples are not necessarily limited thereto.

When a camera and/or a capturing action of the user using the camera is detected in operation <NUM>, the management apparatus may determine that the process is failed in operation <NUM>, and lock the screen in operation <NUM>.

When a camera is not detected in operation <NUM>, the management apparatus may check whether a cell phone is detected in the input image <NUM> using the object detection model <NUM>, in operation <NUM>. Further, similar to operation <NUM>, the management apparatus may check (determine) whether a capturing action of the user corresponding to the cell phone is detected in the input image <NUM>, in operation <NUM>. The management apparatus may determine whether a capturing action of the user is detected based on whether at least one of a pose of the user, a gesture of the user, a position of the capturing device, or consecutive motions of the user corresponding to the cell phone corresponds to a motion suspected as a capturing action.

When a cell phone and/or a capturing action of the user using the cell phone is detected in operation <NUM>, the management apparatus may determine that the process is failed in operation <NUM>, and lock the screen in operation <NUM>.

When a cell phone is not detected in operation <NUM>, the management apparatus determines that the process is successful in operation <NUM>, and perform face recognition in operation <NUM>.

In operation <NUM>, the management apparatus may verify whether a region of at least one portion of a body of the detected user in the input image <NUM> is a face region of an authorized user (e.g., the user A).

<FIG> illustrates an example of a process of detecting multiple users by a management apparatus. Referring to <FIG>, a process of performing a user verification by a management apparatus by detecting a face region from an input image <NUM> including two users is illustrated.

When an input image <NUM> is received, the management apparatus may perform face detection, in operation <NUM>. In operation <NUM>, the management apparatus may detect a face region in the input image <NUM>.

In operation <NUM>, the management apparatus may determine whether the number of face regions detected in operation <NUM> is "<NUM>". When it is determined in operation <NUM> that the number of face regions is not "<NUM>", the management apparatus may determine whether the number of face regions is "<NUM>", in operation <NUM>. When it is determined in operation <NUM> that the number of face regions is "<NUM>", the management apparatus may not perform a verification for the user since a face region is not detected. Accordingly, the management apparatus may determine that the process is failed and lock a screen in operation <NUM>. In addition, when it is determined in operation <NUM> that the number of face regions is not "<NUM>", it means that the number of face regions is greater than or equal to "<NUM>", in other words, that an authorized user is with an unauthorized user. Thus, the management apparatus may determine that the process is failed and lock the screen in operation <NUM>.

Conversely, when it is determined in operation <NUM> that the number of face regions is "<NUM>", the management apparatus may detect landmarks from the face region detected in operation <NUM>, in operation <NUM>. "Landmarks" may refer to points, in the input image <NUM>, that have remarkable color or brightness changes and thus may express features of the whole image. The landmarks may also be referred to as "feature points".

In operation <NUM>, the management apparatus may extract features of the face region in the input image <NUM> according to the landmarks detected in operation <NUM>. In operation <NUM>, the management apparatus may extract the features for example, in the form of a feature vector or a feature map. However, examples are not necessarily limited thereto.

In operation <NUM>, the management apparatus may perform a feature verification by the features extracted in operation <NUM>. When the features extracted in operation <NUM> match features of the authorized user stored in an enrollment DB, the management apparatus may determine that the verification is successful in operation <NUM>, and update a template by the input image <NUM> in operation <NUM>.

When the features extracted in operation <NUM> do not match the features of the authorized user stored in the enrollment DB, the management apparatus may determine that the verification is failed in operation <NUM>, and lock the screen in operation <NUM>.

In operation <NUM>, the management apparatus receives an input image including a region of at least one portion of a body of a user.

In operation <NUM>, the management apparatus may determine whether the user is multiple users or a single user using the region of the at least one portion of the body.

In operation <NUM>, the management apparatus may perform a verification for the user by a face region in the input image in response to the determination of operation <NUM> (e.g., the determination that the user is a single user).

In operation <NUM>, the management apparatus may determine whether the input image is a real image or a spoofed image based on whether the verification performed in operation <NUM> is successful.

In operation <NUM>, the management apparatus may allow an access authority to the system to the user, in response to the determination of operation <NUM> (e.g., the determination that the input image is a real image).

In operation <NUM>, the management apparatus may determine whether an abnormal situation occurs in a second input image received after the access authority is allowed in operation <NUM>. Here, the "abnormal situation" may be construed as a situation in which it is determined that an authorized user and/or an unauthorized user attempts to leak information from the system. The abnormal situation may include, for example, a situation in which at least one of another user other than the authorized user, a preset type of capturing device, or a capturing action using a capturing device is detected in the second input image. However, examples are not necessarily limited thereto.

In operation <NUM>, the management apparatus may determine whether an unauthorized user is detected based on the number of face regions included in the second input image received after the access authority is allowed. When a plurality of face regions are detected in the second input image, the management apparatus may determine that another user (e.g., an unauthorized user) is detected. When an unauthorized user is detected, the management apparatus may determine that an abnormal situation occurs.

In operation <NUM>, the management apparatus determines whether an object corresponding to the preset type of capturing device is detected in the second input image. Alternatively or additionally, the management apparatus may determine whether a capturing action of the user corresponding to the preset type of capturing device is detected in the second input image. For example, the management apparatus may determine whether a capturing action of the user is detected, based on whether at least one of a pose of the user facing a screen provided through the system, a gesture of the user, a position of the capturing device corresponding to the screen, or a motion of the user using the capturing device in the second input image corresponds to a motion suspected as a capturing action.

In operation <NUM>, the management apparatus may block the access authority of the user to the system in response to the determination of operation <NUM> (e.g., the determination that an abnormal situation occurs).

<FIG> illustrates an example of an apparatus for managing an access authority. Referring to <FIG>, a management apparatus <NUM> may include a communication interface <NUM>, a processor <NUM> (e.g., one or more processors), a memory <NUM> (e.g., one or more memories), and a sensor <NUM> (e.g., one or more sensors). The communication interface <NUM>, the processor <NUM>, and the memory <NUM> may be connected to each other through a communication bus <NUM>.

The communication interface <NUM> may receive (e.g., from the sensor <NUM>) an input image including at least one portion of a body of a user. The input image may be, for example, an image captured by the sensor <NUM>, an image captured by an image sensor in a terminal of the user who is telecommuting, or an image captured by a CCTV, or a camera in a laptop computer or a desktop computer provided in a place of work of the user. The sensor <NUM> may be or include an image sensor such as a camera that captures the input image.

The communication interface <NUM> may transmit, to an outside of the management apparatus <NUM>, a result of determining whether the user included in the input image is multiple users, a result of performing a verification for the user, and a result of determining whether the input image is a real image, performed by the processor <NUM>.

The processor <NUM> may determine whether the user is multiple users or a single user using a region of the at least one portion of the body included in the input image received through the communication interface <NUM>. The processor <NUM> may perform a verification for the user by a face region in the input image, in response to the determination that the user is a single user. The processor <NUM> may determine whether the input image is a real image or a spoofed image based on whether the verification is successful. The processor <NUM> may allow an access authority to the system to the user, in response to the determination that the input image is a real image.

Alternatively or additionally, according to an example, the processor <NUM> may determine whether the input image is a real image or a spoofed image using the face region in the input image, in response to the determination that the user is a single user. In this case, the processor <NUM> may perform a verification for the user by the face region, in response to the determination that the input image is a real image. The processor <NUM> may allow an access authority to the system to the user based on whether the verification is successful.

The processor <NUM> may execute a program and control the management apparatus <NUM>. Program codes to be executed by the processor <NUM> may be stored in the memory <NUM>.

The memory <NUM> may store the input image received from the communication interface <NUM>. Further, the memory <NUM> may store a variety of information generated in the processing process of the processor <NUM> described above. In addition, the memory <NUM> may store a variety of data and programs. The memory <NUM> may include a volatile memory or a non-volatile memory. The memory <NUM> may include a large-capacity storage medium such as a hard disk to store the variety of data.

The management apparatus <NUM> may be, for example, a telecommuting security system or a device that performs similar functions thereto.

In addition, the processor <NUM> may perform the at least one method described with reference to <FIG> or a scheme corresponding to the at least one method. The processor <NUM> may be a management apparatus implemented by hardware including a circuit having a physical structure to perform desired operations. For example, the desired operations may include code or instructions included in a program. The hardware-implemented management apparatus <NUM> may include, for example, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and a neural processing unit (NPU).

The management apparatuses, enrollment DBs, communication interfaces, processors, memories, sensors, communication buses, management apparatus <NUM>, enrollment DB <NUM>, enrollment DB <NUM>, enrollment DB <NUM>, management apparatus <NUM>, communication interface <NUM>, processor <NUM>, memory <NUM>, sensor <NUM>, communication bus <NUM>, and other apparatuses, devices, units, modules, and components described herein with respect to <FIG> are implemented by or representative of hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

Claim 1:
A method with access authority management, the method comprising:
receiving an input image comprising a region of at least one portion of a body of a user;
determining whether the user corresponds to multiple users or a single user using the region of the at least one portion of the body;
performing a verification for the user based on a face region in the input image, in response to the determination that the user is the single user;
determining whether the input image is a real image or a spoofed image based on whether the verification is successful;
allowing an access authority to a system to the user, in response to the determination that the input image is the real image; and
characterized by further comprising:
determining whether a preset type of capturing device is detected in the input image; and
blocking an access authority of the user to the system in response to the determination that the preset type of capturing device is detected.