Patent ID: 12189752

DETAILED DESCRIPTION

The following detailed description is made with reference to the accompanying drawings and is provided to assist in a comprehensive understanding of various example embodiments of the present disclosure. The following description includes various details to assist in that understanding, but these are to be regarded merely as examples and not for the purpose of limiting the present disclosure as defined by the appended claims and their equivalents. The words and phrases used in the following description are merely used to enable a clear and consistent understanding of the present disclosure. In addition, descriptions of well-known structures, functions, and configurations may have been omitted for clarity and conciseness. Those of ordinary skill in the art will recognize that various changes and modifications of the examples described herein can be made without departing from the spirit and scope of the present disclosure.

FIG.1is a schematic diagram of a multimodal authentication system100, according to one or more aspects of the present disclosure. It should be appreciated that various example embodiments of inventive concepts disclosed herein are not limited to specific numbers or combinations of devices, and there may be one or multiple devices in the multimodal authentication system100, which may itself consist of multiple communication networks and various known or future developed wireless connectivity technologies, protocols, devices, and the like. The multimodal authentication system100can include a user180in proximity101of a user interface device170coupled via connection103to a multimodal authentication device160that can access a network resource150via a connection105.

A user180within the multimodal authentication system100may request access to one or more devices, one or more resources, and/or one or more services, for example, one or more resources and/or one or more services provided by a network resource150, a multimodal authentication device160, any other device, or a combination thereof, that require a multimodal authentication by a multimodal authentication device160. The multimodal authentication device160can include one or more elements including, but not limited to, any of an output device102, a network interface104, a power supply106, a controller108, a memory110, a user interface114, any other element, or a combination thereof.

The output device102can comprise any device that provides for the playback and/or reproduction of any of audio, video, multimedia content, any other content, or any combination thereof, for example, to a user180within proximity101including, but not limited to, any of a speaker, a sound bar, a display (such as a television, a monitor, a projector, and/or any other audio playback device, video playback device, or both), any other device that is capable of providing multi-media content for consumption by a user180, or any combination thereof. For example, output device102can output an audio authentication command, a visual and/or audio liveliness authentication scheme.

The network interface104can include, but is not limited to, various network cards, interfaces, and circuitry implemented in software and/or hardware to enable communications with any of one or more elements of user interface device170, network resource150, any other device, or a combination thereof using the communication protocol(s) in accordance with any connection, for example, connection103and105. The power supply106supplies power to any one or more of the internal elements of the multimodal authentication device160, for example, through an internal bus. The power supply106can be a self-contained power source such as a battery pack with an interface to be powered through an electrical charger connected to an outlet (for example, either directly or by way of another device). The power supply106can also include a rechargeable battery that can be detached allowing for replacement such as a nickel-cadmium (NiCd), nickel metal hydride (NiMH), a lithium-ion (Li-ion), or a lithium Polymer (Li-pol) battery.

The controller208controls the general operations of the multimodal authentication device160and can comprise any of or any combination of a central processing unit (CPU), a hardware microprocessor, a hardware processor, a multi-core processor, a single core processor, a field programmable gate array (FPGA), a microcontroller, an application specific integrated circuit (ASIC), a digital signal processor (DSP), or other similar processing device capable of executing any type of computer-readable instructions, algorithms, or software including the software112stored in memory110for controlling the operation and functions of the multimodal authentication device160in accordance with the embodiments described in the present disclosure. Communication between any of the element (for example, elements102,104,106,110,112, and/or114) of the multimodal authentication device160can be established using an internal bus.

The memory110can comprise a single memory or one or more memories or memory locations that can include, but are not limited to, any of a random access memory (RAM), a dynamic random access memory (DRAM) a memory buffer, a hard drive, a database, an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), a flash memory, logic blocks of a field programmable gate array (FPGA), an optical drive, a hard disk or any other various layers of memory hierarchy. The memory110can be used to store any type of computer-readable instructions, software, or algorithms including software112for controlling the general function and operations of the multimodal authentication device160in accordance with the embodiments described in the present disclosure. In one or more embodiments, software112includes one or more applications and/or computer-readable instructions for providing multimodal authentication.

The user interface114can comprise any of one or more tactile inputs (for example, a push button, a selector, a dial, etc.), a camera, a keyboard, an audio input, for example, a microphone, a keypad, a liquid crystal display (LCD), a thin film transistor (TFT), a light-emitting diode (LED), a high definition (HD) or other similar display device including a display device having touch screen capabilities so as to allow interaction between one or more users180and the multimodal authentication device160, or a combination thereof.

In one or more embodiments, the multimodal authentication device is coupled or connected to a user interface device170via a connection103so as to provide and/or receive audio and/or visual inputs and/or outputs to and/or from a user180within a proximity101. In one or more embodiments, the user interface device170or one or more elements of the user interface device170are incorporated within or as part of the multimodal authentication device160. The user interface device170can comprise an audio capture device172, an audio output device174, an image capture device176, a display device178, any other element, or any combination thereof.

The audio capture device172can, for example, comprise a microphone or any other device that can receive and/or record audio or sound, for example, a voice of user180. The audio output device174can be similar to or the same as one or more components of the output device102. The audio output device174can comprise, for example, a speaker or any other device that can output an audio or a sound. The image capture device176can, for example, comprise a camera or any other device that can receive and/or record an image. The display device178can be similar to or the same as one or more components of the output device102. The display device178can comprise, for example, a display or any other device that can output an image and/or content.

In one or more embodiments any of connection103or connection105can be a bidirectional communication link such that any one or more communications or messages can be sent and/or received by any of the multimodal authentication device160, the multimodal authentication device160, the user interface device170, or any combination thereof. A connection103or a connection105can be a wired and/or wireless connection.

Further, any, all, or some of the electronic elements or electronic computing devices can be adapted to execute any operating system, including Linux, UNIX, Windows, MacOS, DOS, and ChromOS as well as virtual machines adapted to virtualize execution of a particular operating system, including customized and proprietary operating systems. Any, all or some of the electronic components or electronic computing devices are further equipped with components to facilitate communication with other devices over the one or more network connections to local and wide area networks, wireless and wired networks, public and private networks, and any other communication network enabling communication in the multimodal authentication system100.

FIG.2is a block diagram of a multimodal authentication device160, according to one or more aspects of the present disclosure. In one or more embodiments, a multimodal authentication device160can comprise and/or be connected to a user interface device170and/or one or more elements of a user interface device170. While the elements172,174,176and178are illustrated as separate elements, the present disclosure contemplates that any one or more of these elements can be integrated as a single element. A display device178(such as a television) can display one or more images at one or more locations such as one or more locations L11 through Lcr, where c represents column number c and r represents row number where a display device178can comprise any number of rows and columns. The multimodal authentication device160can cause one or more images, for example, any one or more of images300A-300n(collectively referred to as image(s)300) ofFIG.3, where n represents any number of images300, to be displayed at any one or more locations L11 through Lcr. The one or more images300can be stored in a memory110, received or pulled from a network resource150, or both. The display of an image300can be used by the multimodal authentication device160to authenticate a user as part of, for example, any of an audio authentication, an audio liveliness authentication, a visual authentication, a visual liveliness authentication, or any combination thereof. For example, the one or more images300can be selected based on any one or more factors, such as any of an ease of pronunciation of the corresponding identifier associated with the image300(such as illustrated in Table 1), number of syllables of the corresponding identifier with the image300, familiarity or popularity of the image300(such as international symbols), ease of depiction or clarity of the image300, selected language for the authentication (such as English, Spanish, Mandarin, German, etc.), age or age range of a user, educational level of a user, any other factor, or any combination thereof.

TABLE 1ImagedIdentifier300AFlower300BCat300CApple300nDog

FIGS.4A-4Cillustrate an exemplary liveliness authentication scheme, according to one or more aspects of the present disclosure. As an example of a liveliness authentication scheme, a multimodal authentication device160can cause an image300A to be initially displayed at a location L11 (such as location row 1, column 1), for example, of a display device178as illustrated inFIG.4A. The multimodal authentication device160can instruct a user, such as user180, to vocally identify or provide any other auditory response, for example, by outputting audio via the audio output deice174, the image300A. The image300A can be transitioned to a location L67 (such as location column 6, row 7) as illustrated inFIG.4Band then to a location L24 (such as location column 2, row 4) as illustrated inFIG.4C. At each location, the multimodal authentication device160can receive and/or record the audio from the user180, for example, via an audio capture device172so as to determine an audio authentication, an audio liveliness authentication, or both. The multimodal authentication device160can also track or record any facial orientation, for example, an eye movement (such as an eye transition or a gaze), a facial angle, a facial tilt, any other facial positioning, or any combination thereof of a user180, for example, via an input received from an image capture deice176so as to determine a visual authentication, a visual liveliness authentication, or both.

FIGS.5A-5Dillustrate an exemplary liveliness authentication scheme, according to one or more aspects of the present disclosure. As discussed with reference toFIGS.4A-4C, the multimodal authentication device160can track facial orientation of a user180within proximity of the multimodal authentication device160, a user interface device170, or both. For example, a user180can initially have a gaze (eye positioning) looking straight towards an object, such as illustrated inFIGS.5A and5Bor at an angle, such as illustrated inFIGS.5C and5D. As an example, an image300can be displayed at a first location with a first facial orientation of the user180ofFIG.5Aand at a second location with a second facial orientation of the user180ofFIG.5C. As an example, an image300can be displayed at a third location with a third facial orientation of the user180ofFIG.5Band at a fourth location with a fourth facial orientation of the user180ofFIG.5D. In one or more embodiments, the facial orientation of the user180can change, for example, from an angle ofFIG.5Ato the angle ofFIG.5D,FIG.5BtoFIG.5C, or any other angle or orientation. The gaze and/or orientation can be utilized by the multimodal authentication device to determine any of an audio authentication, an audio liveliness authentication, a visual authentication, a visual liveliness authentication, or any combination thereof.

FIG.6illustrates an exemplary liveliness authentication scheme, according to one or more aspects of the present disclosure. A user180can be within proximity of a multimodal authentication device160and receive an instruction to vocally identify an image300. The vocal identification can be received or recorded by the multimodal authentication device160. For example, the vocalization (the audio) by the user180can be captured (such as received or recorded) by an audio capture device172as an analog signal160while the lip movement can be captured (such as received or recorded) by an image capture device176. The captured audio and video can be used by the multimodal authentication device160to determine any of an audio authentication, an audio liveliness authentication, a visual authentication, a visual liveliness authentication, or any combination thereof.

FIG.7illustrates a flow diagram of providing a multimodal authentication by a multimodal authentication device160, according to one or more aspects of the present disclosure. While the steps702-723are presented in a certain order, the present disclosure contemplates that any one or more steps can be performed simultaneously, substantially simultaneously, repeatedly, in any order or not at all (omitted).

At step702, a user180within a proximity100of a multimodal authentication device160, a user interface device170, or both is provided instructions for a liveliness authentication scheme. For example, a user can be within proximity of a television and be instructed visually, auditorily or both on how to proceed with authentication so that the user has access to one or more devices, one or more services, and/or one or more resources associated with a device, such as any of a multimodal authentication device160, a network resource150, any other device, or a combination thereof. The instructions can provide the user180with the requirements of the liveliness authentication scheme, for example, that the user180must identify verbally or orally (say aloud) a displayed image300.

At step703, a location random number series RL1through RLmand an image (such as a picture) random number series RP1through RPnare generated where m and n represent any value. RL represents a location for display of an image300and RP represents an image300selected from a plurality of images300. For example, RP1can correspond to a first image300A displayed at RL1that can correspond to a first location L11, RP2can correspond to a second image300B displayed at RL2that can correspond to a second location L67, and RP3can correspond to a third image300ndisplayed at RLnthat can correspond to a third location L24. In this way, not only are the images300selected randomly but the location for display of the images300are randomly selected. In one or more embodiments, only a single image is selected for display at various locations, no two identical images are displayed at various locations, or any combination of images can be displayed at the various locations.

At step704, initial variables required for determination a multimodal authentication are initialized or set. For example, one or more counters are set to a predetermined value such as a “1” or a “0”. An initial counter (i) is set to, for example, a “1”. An initial result of a video authentication (RVface), an initial result of a video liveliness authentication (RVlive), an initial result of an audio of a voice authentication (RAvoice), and an initial value of a result of an audio liveliness authentication (RAlive) are each set to, for example, a “0”.

At step706, the multimodal authentication device160displays a selected image (P(RPi) at a location on a display device (L(RLi), for example, as illustrated inFIGS.4A-4C. At step707, as the selected image is displayed a user's gaze and/or orientation is captured, for example, an audio input and/or a video input is received by the multimodal authentication device160. For example, the instructions at step701can indicate that a user is to speak the identifier associated with the displayed image, such as illustrated in Table 1. The multimodal authentication device160captures the angle, the orientation, and/or the audio from the user.

At step708, the multimodal authentication device160determines if the user is authenticated based on a visual data, for example, based on a facial recognition algorithm. In one or more embodiments, the captured or received video data is compared to a stored image of a user. If the user is visually authenticated, the process continues at step709where the variable RVface is incremented. The process continues at step712, where a visual liveliness authentication is determined, for example, multimodal authentication device160can determine if the user's face, gaze, and/or orientation is as expected. If a visual liveliness authentication is determined, at step713the variable RVlive is incremented. The process continues at step714where the multimodal authentication device160determines if the captured audio is authenticated, for example, by comparing a received vocal or audio input to a previously stored or retrieved audio input associated with the user. If the audio input is authenticated, then at step715the variable RAvoice is incremented. The process continues at step716where the multimodal authentication device160determines if an audio liveliness is authenticated. For example, one or more video and audio inputs are processed such that the one or more timestamps associated with the start of the user's lip movement from the captured video and one or more timestamps associated with the start of a command from the captured audio are compared to determine if the timestamps match. As an example, the timestamp of the lip movements is compared with the timestamp that one or more vowels are spoken and checked to determine if a match exists. In this way, an audio liveliness authentication is determined. If the audio voice liveliness is authenticated, the variable RAlive is incremented.

The process continues at step718where the variable i (such as an iteration counter) is incremented. At step719it is determined if the number of iterations through the process (i) has reached the number of image/location combinations (n). The process proceeds to step706if additional image/location combinations remain and to step721otherwise. At step721, the multimodal authentication device160compares the variables to a threshold value, for example, to determine if the variable have reached and/or exceeded a threshold value associated with the values. For example, RVface is compared to an RVface threshold (RVfaceThr), RVlive is compared to an RVlive threshold (RVliveThr), RAvoice is compared to a RAvoice threshold (RAVoiceThr) and RAlive is compared to an RAlive threshold (RAlive). If through the number of iterations (i) the variables do not meet a threshold standard, the process continues at step723where the user is not authenticated and thus is not permitted access to one or more devices, one or more services, and/or one or more resources. Otherwise, the process continues at step722where the user is authenticated and allowed to access one or more services or resources. In one or more embodiments, the user is informed of the authentication visually, auditorily, or both, for example, via the user interface device170, an out device102, a user interface114, or any combination thereof.

FIG.8illustrates a flow diagram of a method of a multimodal authentication device170for providing multimodal authentication, according to one or more aspects of the present disclosure. InFIG.8, it is assumed that any one or more of the devices include their respective controllers and their respective software stored in their respective memories, as discussed above in connection with any ofFIGS.1-7, which when executed by their respective controllers perform the functions and operations in accordance with the example embodiments of the present disclosure (for example, including performing a configuration of one or more network devices). While the steps S602-S612are presented in a certain order, the present disclosure contemplates that any one or more steps can be performed simultaneously, substantially simultaneously, repeatedly, in any order or not at all (omitted).

At step S602, a multimodal authentication device160provides a multimodal authentication that includes a liveliness determination by displaying one or more images, such as one or more images stored in a memory110or received from a network resources150, at one or more locations of a display device178associated with the multimodal authentication device160. In one or more embodiments, the display device178is integrated within or as part of the multimodal authentication device160. In one or more embodiments, a location random number series is generated with the one or more locations being based on the location random number series and an image random number series is generated with the one or more images for display at the one or more locations being selected based on the image random number series. In one or more embodiments, prior to displaying the one or more images, the multimodal authentication device160provides one or more instructions at the display device178, an audio output device174, or both. In response to the instructions, the multimodal authentication device160can receive a user input wherein displaying the one or more images is based on the user input. For example, the user input can be associated with a language, a genre, an education level, an age level, any other criteria, or any combination thereof.

At step S604, the multimodal authentication device160receives one or more user images of a user180in response to the display of the one or more images at step S602. The user image is received from an image capture device176that is associated with the multimodal authentication device. In one or more embodiments, the image capture device176is integrated within or as part of the multimodal authentication device160.

At step S606, the multimodal authentication device160receives one or more audio inputs in response to the display of the one or more images at step S602. The user audio is received from an audio capture device172associated with the multimodal authentication device160. In one or more embodiments, the audio capture device172is integrated within or as part of the multimodal authentication device160.

At step S608, the multimodal authentication device160determines a visual authentication based on the one or more user images. A visual authentication result can be set based on the visual authentication. At step S610, the multimodal authentication device160determines a visual liveliness authentication based on the one or more user images. The determining the visual liveliness authentication can comprise at least one of determining a face angle associated with the one or more user images and determining a gaze angle associated with the one or more user images. The determining the visual liveliness authentication can further comprise at least one of determining that the face angle tracks the displaying of the one or more images at each of the one or more locations, and determining that the gaze angle tracks the displaying of the one or more images at each of the one or more locations. A visual liveliness authentication result can be set based on the visual liveliness authentication. At step S612, the multimodal authentication device160determines an audio authentication based on the one or more user audio inputs. An audio authentication result can be set based on the audio authentication. At step S612, the multimodal authentication device160determines an audio liveliness authentication based on the one or more audio inputs. An audio liveliness authentication result can be set based on the audio liveliness authentication.

At step S616, the multimodal authentication device160provides a multimodal authentication based on the visual authentication, the visual liveliness authentication, the audio authentication, and the audio liveliness authentication. The multimodal authentication can comprise or be based on each of comparing the visual authentication result to a visual authentication result threshold, comparing the visual liveliness authentication result to a visual liveliness authentication threshold, comparing the audio authentication result to an audio authentication result threshold, and comparing the audio liveliness authentication result to an audio liveliness authentication result threshold.

According to some example embodiments of inventive concepts disclosed herein, there are provided novel solutions for providing a multimodal authentication by a multimodal authentication device that relies on captured video and audio across a series of image/location combinations. The multimodal authentication device provides a significant improvement over traditional systems as the novel multimodal authentication device not only uses captured audio and video but also determines a liveliness associated with the audio and video. By providing such enhanced authentications, security risks associated with a false authentication are reduced or eliminated. For example, home automation systems, voice assistant devices, telehealth systems, etc. are improved by providing a reliable authentication.

Each of the elements of the present invention may be configured by implementing dedicated hardware or a software program on a memory controlling a processor to perform the functions of any of the components or combinations thereof. Any of the components may be implemented as a CPU or other processor reading and executing a software program from a recording medium such as a hard disk or a semiconductor memory, for example. The processes disclosed above constitute examples of algorithms that can be affected by software, applications (apps, or mobile apps), or computer programs. The software, applications, computer programs or algorithms can be stored on a non-transitory computer-readable medium for instructing a computer, such as a processor in an electronic apparatus, to execute the methods or algorithms described herein and shown in the drawing figures. The software and computer programs, which can also be referred to as programs, applications, components, or code, include machine instructions for a programmable processor, and can be implemented in a high-level procedural language, an object-oriented programming language, a functional programming language, a logical programming language, or an assembly language or machine language.

The term “non-transitory computer-readable medium” refers to any computer program product, apparatus or device, such as a magnetic disk, optical disk, solid-state storage device (SSD), memory, and programmable logic devices (PLDs), used to provide machine instructions or data to a programmable data processor, including a computer-readable medium that receives machine instructions as a computer-readable signal. By way of example, a computer-readable medium can comprise DRAM, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired computer-readable program code in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Disk or disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc. Combinations of the above are also included within the scope of computer-readable media.

The word “comprise” or a derivative thereof, when used in a claim, is used in a nonexclusive sense that is not intended to exclude the presence of other elements or steps in a claimed structure or method. As used in the description herein and throughout the claims that follow, “a”, “an”, and “the” includes plural references unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise. Use of the phrases “capable of,” “configured to,” or “operable to” in one or more embodiments refers to some apparatus, logic, hardware, and/or element designed in such a way to enable use thereof in a specified manner.

While the principles of the inventive concepts have been described above in connection with specific devices, apparatuses, systems, algorithms, programs and/or methods, it is to be clearly understood that this description is made only by way of example and not as limitation. The above description illustrates various example embodiments along with examples of how aspects of particular embodiments may be implemented and are presented to illustrate the flexibility and advantages of particular embodiments as defined by the following claims, and should not be deemed to be the only embodiments. One of ordinary skill in the art will appreciate that based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents may be employed without departing from the scope hereof as defined by the claims. It is contemplated that the implementation of the components and functions of the present disclosure can be done with any newly arising technology that may replace any of the above-implemented technologies. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present invention. The benefits, advantages, solutions to problems, and any element(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential features or elements of any or all the claims. The invention is defined solely by the appended claims including any amendments made during the pendency of this application and all equivalents of those claims as issued.