Dynamically changing audio keywords

Aspects of the present invention disclose a method, computer program product, and system for managing a device utilizing user audio communications. The method includes one or more processors identifying an audio communication from a user. The method further includes, responsive to determining that the audio communication from the user does not include an activation command for a listening device, one or more processors determining a location of the user and information associated with the determined location of the user. The method further includes one or more processors determining an action to initiate on a listening device based on the audio communication, the determined information associated with the location of the user, and user preferences. The method further includes one or more processors initiating execution of the determined action on the listening device.

BACKGROUND

The present invention relates generally to the field of natural language processing, and more particularly to computational linguistics.

Speech recognition is a sub-field of computational linguistics that develop methodologies and technologies that enable recognition and translation of spoken language into text by computers. It is also known as “automatic speech recognition” (ASR), “computer speech recognition,” or just “speech to text” (STT). Some speech recognition systems require “training” (also called “enrollment”) where an individual speaker reads text or isolated vocabulary into the system. The system analyzes the person's specific voice and uses it to fine-tune the recognition of that person's speech, resulting in increased accuracy. Systems that do not use training are called “speaker independent” systems. Systems that use training are called “speaker dependent.”

Speech recognition applications include voice user interfaces, such as voice dialing (e.g., “Call home”), call routing (e.g., “I would like to make a collect call”), domotic appliance control, search (e.g., find a podcast where particular words were spoken), simple data entry (e.g., entering a credit card number), preparation of structured documents (e.g., a radiology report), speech-to-text processing (e.g., word processors or emails), and aircraft (usually termed Direct Voice Input).

The term voice recognition or speaker identification refers to identifying the speaker, rather than what they are saying. Recognizing the speaker can simplify the task of translating speech in systems that have been trained on a specific person's voice, or it can be used to authenticate or verify the identity of a speaker as part of a security process.

SUMMARY

Aspects of the present invention disclose a method, computer program product, and system for managing a device utilizing user audio communications. The method includes one or more processors identifying an audio communication from a user. The method further includes, responsive to determining that the audio communication from the user does not include an activation command for a listening device, one or more processors determining a location of the user and information associated with the determined location of the user. The method further includes one or more processors determining an action to initiate on a listening device based on the audio communication, the determined information associated with the location of the user, and user preferences. The method further includes one or more processors initiating execution of the determined action on the listening device.

DETAILED DESCRIPTION

Embodiments of the present invention recognize that the performance of speech recognition systems is a function of processing logic relating to a computing device receiving a wake command and validation of the voice. Embodiments of the present invention recognize that speech recognition by machine is a very complex problem. Speech recognition by machine is complicated by the fact that vocalizations vary in terms of accent, pronunciation, articulation, roughness, nasality, pitch, volume, and speed. Speech is distorted by a background noise and echoes, electrical characteristics.

Embodiments of the present invention recognize that the accuracy of speech recognition by machine may vary with vocabulary size and confusability, speaker dependence versus independence, and isolated, discontinuous or continuous speech, task and language constraints, read versus spontaneous speech, and adverse conditions.

Embodiments of the present invention recognize that current approaches to speech recognition by machine use activation words and the same activation word that is recognized by a machine can be spoken in an audio or video broadcast, or by non-owners of the machine in the same room. Activation word speech recognition by machines can cause machines in audience homes and offices to start listening for input inappropriately, or possibly cause the machine to take an unwanted action. Computing devices that require a wake command by the primary user will not operate without a wake command word being recognized by the computing device.

Embodiments of the present invention provide a method that allows the computing device to determine whether a computing device bypasses the wake command machine speech recognition requirement in the computing device.

Example embodiments, in accordance with the present invention, will now be described in detail with reference to the Figures.FIG. 1is a functional block diagram, illustrating distributed data processing environment100. Distributed data processing environment100includes listening computing device110and server120, all interconnected over network185.

In one embodiment, listening computing device110includes graphical user interface (GUI)130, web browser150, storage160, and program200. The various programs on listening computing device110include a web browser, an electronic mail client, security software (e.g., a firewall program, a geo-locating program, an encryption program, etc.), an instant messaging (IM) application (app), an audio microphone program, and a communication (e.g., phone) application.

Listening computing device110may be a desktop computer, a laptop computer, a tablet computer, a specialized computer server, a smartphone, a listening device, a wearable device (e.g., smart watch, personal fitness device, personal safety device), or any programmable computer system known in the art with an interactive display or any other computer system known in the art. In certain embodiments, listening computing device110represents a computer system utilizing clustered computers and components that act as a single pool of seamless resources when accessed through network185, as is common in data centers and with cloud computing applications. In general, listening computing device110is representative of any programmable electronic device or combination of programmable electronic devices capable of executing machine-readable program instructions and communicating with other computer devices via a network.

In another embodiment, listening computing device110is a cognitive microphone that records the voice of a user and through software located on one or more computing devices, such as listening computing device110, sends the audible word spoken by a user to a voice to text application service located on server120. Listening computing device110is constantly in a sentry type listening mode and is activated by a wake word. A wake word is a word recognized by a speech recognition server as an activation word that enables listening computing device110to begin to process words spoken by a user.

In one embodiment, graphical user interface130operates on listening computing device110. In another embodiment, graphical user interface130operates on another computer in a server-based setting, for example, on a server computer (e.g., server120). In yet another embodiment, graphical user interface130operates on listening computing device110simultaneously with a server computer interconnected through network185(e.g., server120). Graphical user interface130may be any user interface used to access information from listening computing device110, such as information gathered or produced by program200. Additionally, graphical user interface130may be any user interface used to supply information to listening computing device110, such as information supplied by a user to be used by program200. In some embodiments, graphical user interface130may present a generic web browser used to retrieve, present, and negotiate resources from the Internet. In other embodiments, graphical user interface130may be a software or application that enables a user at listening computing device110access to network185.

In yet another embodiment, a user of listening computing device110can interact with graphical user interface130through a touch screen that performs as both an input device to a graphical user interface (GUI) and as an output device (i.e., an electronic display) presenting a plurality of icons associated with software applications or images depicting the executing software application. Optionally, a software application (e.g., a web browser) can generate graphical user interface130operating within the GUI of listening computing device110. Graphical user interface130accepts input from a plurality of input/output (I/O) devices including, but not limited to, a tactile sensor interface (e.g., a touch screen or a touchpad) referred to as a multi-touch display. An I/O device interfacing with graphical user interface130may be connected to listening computing device110, which may operate utilizing wired (e.g., USB port) or wireless network communications (e.g., infrared, NFC, etc.). Listening computing device110may include components, as depicted and described in further detail with respect toFIG. 3, in accordance with embodiments of the present invention.

Web browser150may be a generic web browser used to retrieve, present, and traverse information resources from the Internet. In some embodiments, web browser150may be a web browser designed for a mobile device. In other embodiments, web browser150may be a web browser designed for a traditional computing device, such as a desktop computer, PC, or laptop. In general, web browser150may be any application or software that enables a user of listening computing device110to access a webpage over network185. In the depicted environment, web browser150resides on listening computing device110. In other embodiments, web browser150, or similar web browsers, may reside on other computing devices capable of accessing a webpage over network185.

Storage160(e.g., a database) located on listening computing device110, represents any type of storage device capable of storing data that is accessed and utilized by listening computing device110. In other embodiments, storage160represents multiple storage devices within listening computing device110. Storage160stores information such as, but not limited to, account information, credentials for authentication, user preferences, lists of preferred users, previously visited websites, history of visited Wi-Fi portals, and the history of the location of the computing device. Storage160contains data utilized by listening computing device110. In an example, storage160contains a history of wake commands utilized by listening computing device110. In another example, storage160contains data of user configurations of listening computing device110. In this example, a user is capable of storing data of user preferences for the operational settings of listening computing device110.

Geographic information system (GIS)180is a system designed to capture, store, manipulate, analyze, manage, and present all types of geographic data. In general, GIS180describes any information system that integrates, stores, edits, analyzes, shares, and displays geographic information. Listening computing device110can allow server120, through a handshake, to remotely access GIS180on listening computing device110. In some embodiments, GIS180, on listening computing device110, contains GPS receivers. In other embodiments, GIS180contains radio-frequency identification (RFID) tags. In still other embodiments, GIS180contains other sensors and/or receivers used by program200to determine the location of listening computing device110. GIS180may utilize GPS or triangulation techniques to determine the physical location of listening computing device110.

In another example, GIS180may utilize a series of RFID tag readers to locate a server with an RFID tag. GIS180contains a globally unique identifier (GUID) that is assigned for each computing device. The GUID can be the MAC address for the device. The device is tracked through the GUID located on GIS180.

GIS180operates to enhance program200to increase the accuracy or amount of location-related information that program200obtains. GIS180utilizes GUID information, combined with a sampling of data collected by GIS180, to make determinations regarding the location of listening computing device110. In an example embodiment, GIS180may receive a series of data points collected by the GUID. GIS180may use the received data points to create a path and compare that path to GUID information corresponding to the location of the received data points.

Program200is a program, located on listening computing device110, that dynamically changes an audio keyword. In an embodiment, program200includes the subprograms; listening device software115and audio processor117, a sub program of listening device software115.

Listening device software115is a cloud-based sub program, of program200, that allows a user to connect to and control listening computing device110. Listening device software115contains a sub program, audio processor117, that allows listening computing device110to recognize an audible word spoken by a user, and to enable listening computing device110to record the audible word spoken by a user. After listening computing device110, through audio processor117, has recorded the audible word spoken by a user, listening computing device110, through listening device software115, processes the audible word spoken by a user to a voice service located on listening computing device110.

Listening device software115provides the user the ability to interface with listening computing device110and to function as a control center that allows a user to change network connectivity, allows a user to configure date and time, allows a user to set the language listening computing device110recognizes, and the ability to sync listening computing device110with one or more programs located on listening computing device110. Listening device software115allows a user the capability to integrate listening computing device110to one or more programs stored on listening computing device110. In an example, listening device software115allows a user to connect listening computing device110to a music service application located on listening computing device110. In this another example, listening device software115provides the capability for a user to integrate a car ordering service application on listening computing device110with listening device software115.

In general, network185can be any combination of connections and protocols that will support communications among listening computing device110. Network185can include, for example, a local area network (LAN), a wide area network (WAN), such as the Internet, a cellular network, or any combination of the preceding, and can further include wired, wireless, and/or fiber optic connections.

Server120may be a desktop computer, a laptop computer, a tablet computer, a specialized computer server, a smartphone, or any other computer system known in the art. In certain embodiments, server120represents a computer system utilizing clustered computers and components that act as a single pool of seamless resources when accessed through network185, as is common in data centers and with cloud computing applications. In general, server120is representative of any programmable electronic device or combination of programmable electronic devices capable of executing machine-readable program instructions and communicating with other computer devices via a network. In one embodiment, server120includes database170.

Database170may be a repository that may be read by program200. In an example embodiment, database170can include information, such as Voice ID data of a primary user. The primary user stores a pre-recorded voice print in database170that is used to identify the primary user. In some embodiments, database170may access and retrieve information from another database or storage device that has voice print information for other users. In an example, program200may access other computing devices to retrieve data of voice print information of the users of the computing devices. In other embodiments, a program on server120may prompt and update database170with information. The data stored to database170may be changed or updated by data input by a user, such as a user with access to server120. In one embodiment, database170resides on server120. In other embodiments, database170may reside on another server, another computing device, or mobile device, provided that database170is accessible to program200. Database170operates to enhance program200by increasing the accuracy or amount of data available to program200. Database170stores keyword data of typical language between one or more users that would indicate a telephone call, video conference, or live conversation between one or more users.

In an embodiment, server120is capable of initiating a handshake process between server120and listening computing device110. Handshaking is an automated process of negotiation that dynamically sets parameters of a communications channel established between two entities before normal communication over the channel begins. Handshaking follows the physical establishment of the channel and precedes normal information transfer. Handshaking facilitates connecting heterogeneous computing systems, or equipment, over a communication channel without the need for user intervention to set parameters. In an example, server120initiates the handshake process by sending a massage to listening computing device110indicating that server120wants to establish a communication channel in order to gain access to programs on listening computing device110.

FIG. 2is a flowchart depicting operational steps of program200, a program for dynamically changing an audio keyword, in accordance with an embodiment of the present invention.

In step210, program200identifies a user communication. In an embodiment, program200operates in a listening state and awaits a command, request, or question from a user. In an example, program200identifies a spoken command and determines that the command is the ‘wake command.’ In this example, program200receives the ‘wake command’ and activates listening computing device110. In another example, program200receives a user communication and identifies that the command, request, or question is inaudible because program200is unable to recognize the communication.

In decision step220, program200determines whether an audio keyword was detected. In an example, program200identifies the user communication in step210and compares the identified voiceprint to a pre-recorded user wake command voiceprint on storage160or database170to determine if the voiceprint matches. In response to determining that an audio keyword was detected in the user communication (decision step220, “yes” branch), program200processes the audio keyword. In an embodiment, program200determines whether the wake command came from the primary user by verifying the voice print with the voice id on database170. In an example, program200processes the positive wake command detected after verification and activates listening computing device110to emerge from sentry mode (step230).

In step230, program200processes the positive audio keyword detected as verified by the primary user and the normal wake command processing occurs. In an example, program200determines that a command, question, or a request was preceded by the verified user speaking the wake command, and program200activates listening computing device110.

In response to determining that no audio keyword was detected in the user communication (decision step240, “no” branch), program200determines information associated with the location of the user. In an embodiment, program200utilizes GIS180to determine, and acquire, information associated with the location of users in the room through detecting one or more mobile devices. In an example, program200uses GPS information, from GIS180, to determine information associated with the location users in the room in relation to listening computing device110. Program200accesses GIS180to utilize GPS or triangulation techniques to determine, and update, the physical location of one or more mobile devices of one or more users.

In another embodiment, program200determines the number of users in the room via active listening and active awareness of more than one voice print. In an example, program200, through listening computing device110, accesses database170to retrieve data of the Voice ID of the primary user of listening computing device110. In this example, program200determines the information associated with the location of users in the room by analyzing the number of unique voice prints that program200detects. Program200is capable of determining if the primary user is present in the room through analyzing and comparing the detected voice prints with the pre-recorded voice print of the primary user located on database170. Program200is capable of detecting if a primary user is on a phone call through comparing an analysis of the conversation with data of common keywords used in a phone conversation, such as the word “hello” that is not spoken in conjunction with a separate voice print.

In decision step250, program200determines whether to initiate an action based on identified commands. In an embodiment, program200utilizes the information associated with the location of the user, determined in step240, to determine whether to initiate an action. In an example, program200determines (decision step250, “yes” branch) that program200will initiate an action on listening computing device110by utilizing the information from step240. In this example, in step240, program200accesses network activity to search for active user phones, tablets, or other computing devices that contain a log of associated location information. Program200identifies information associated with the location of listening computing device110and verifies the location of listening computing device110by utilizing known Internet protocol address routing mechanisms. In this example, program200determines an action, such as to wake up listening computing device110, based on an identified information from step240verifying that the primary user is the only user present in the environment by analyzing the last known location of the primary user, though GIS180, and by analyzing the location of any other user in the room. Program200determines that no anomalies have been detected in the information that program200identified, and program200determines that program200will perform an action despite the fact that an audio keyword was not detected in decision step220.

In another embodiment, program200utilizes motion sensors, video surveillance, and other technology to determine whether to initiate an action based on the identified location of the user. In an example, a motion sensor, connected to listening computing device110, detects a user in the room and compares the information associated with the location of the user in step240, the voice-print of the primary user to the voice print id located on database170to determine the user to be the primary user. Program200determines, through voice recognition and motion sensors, that no other user is present in the room and program200determines that program200will perform an action, such as processing a request to turn off a music application.

In another embodiment, program200determines that program200will initiate an action based on the identified command utilizing information associated with the location of the user as determined by video surveillance technology. In this example, program200utilizes facial recognition technology to determine that program200will initiate an action based on the information associated with the location of a user. Program200determines whether program200will initiate the action based on the information associated with the location of the user.

In another embodiment, program200is capable of determining whether to initiate an action based on the information associated with the location of the user as determined by eye scan technology. In an example, program200disables the wake command requirement utilizing eye scan technology. In this example, program200is capable of detecting if a user is looking at a specific target, such as listening computing device110and utilizing the information associated with the location of the user as determined from the eye scan. In this example, a user would look at a pre-defined location point, if not listening computing device110, and speak a request, command, or question and program200determines the information associated with the user based on the location of the user. Listening computing device110is activated in response to the request, command, or question without the wake command.

In another embodiment, program200determines that program200is not capable of initiating an action (decision step250, “no” branch) program200was unable to confirm information associated with the location of the user that enables program200to initiate an action based on the location information determined in step240. In an example, program200determines that the primary user is not present in the room. Program200determines that program200cannot initiate an action when the primary user is not in the room and returns to sentry mode and ends the program.

In step260, program200determines the action to perform based on the identified user communication and user preferences. In an embodiment, program200determines an action to perform based on the command given by the primary user and user preferences stored in storage160or database170. In an example, program200activates listening computing device110in response to the request from the user. In this example, program200receives a “turn on some music” command from a user. Program200retrieves information of a history of music selections from the user, and a history of the frequency of music selections from a user. Program200determines that the program200will play a song that program200determines the user to enjoy based on the history of the song being played and the time of day, when the user returns home, that the user typically plays the song.

In another example, program200determines an action to perform based on a question identified from the user. In this example, program200determines an action to perform based on the fact that more than one person is in the room. Program200identifies that the user has requested the user's credit card information. In this example, program200challenges the primary user to provide further authentication of the identity of the primary user in addition to the voice print because an anomaly was detected after the processing logic of determining information associated with the location of the user was performed in step240. In this example, program200determines, in light of the environment determined by program200, that program200will perform an action that limits the amount of sensitive information broadcasted over audio.

In another example, program200determines action to perform based on a request from the user. In this example, program200identifies a request from a user to “turn on the heat.” Program200retrieves a history of temperature selections from a user that correspond to the temperature fluctuations outside and determines an action to perform based on the request from the user in combination with a history of information related to the request stored to storage160or database170.

In step270, program200executes the action. In an embodiment, program200executes a command from a user. In an example, program200receives an “turn on some music” command from a user. Program200activates a music application and determines a song to play based on an analysis of a history of musical selections from the user. In this example, program200may present a user with more than one choice of music based on user preferences and a history of a selection of music from the user.

In another example, program200executes an action based on a request from a user. In this example, program200identifies a request from the user to “change the credit card information” in an online e-commerce user profile. Program200, based on the action determined in step260that authenticates the user, accessed storage160or database170and changes the default credit card based on the user's frequent use of the credit card.

In another example, program200identifies a command from a user to “turn on the heat” in the home. Program200, based on the information determined in step260, activates the heat at a setting of seventy degrees based on user preferences.

Server120includes communications fabric302, which provides communications between cache316, memory306, persistent storage308, network adapter310, and input/output (I/O) interface(s)312. Communications fabric302can be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system. For example, communications fabric302can be implemented with one or more buses or a crossbar switch.

Memory306and persistent storage308are computer readable storage media. In this embodiment, memory306includes random access memory (RAM). In general, memory306can include any suitable volatile or non-volatile computer readable storage media. Cache316is a fast memory that enhances the performance of computer processor(s)304by holding recently accessed data, and data near accessed data, from memory306.

Program200may be stored in persistent storage308and in memory306for execution by one or more of the respective computer processors304via cache316. In an embodiment, persistent storage308includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage308can include a solid state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer readable storage media that is capable of storing program instructions or digital information.

Network adapter310, in these examples, provides for communications with other data processing systems or devices. In these examples, network adapter310includes one or more network interface cards. Network adapter310may provide communications through the use of either or both physical and wireless communications links. Program200may be downloaded to persistent storage308through network adapter310.

I/O interface(s)312allows for input and output of data with other devices that may be connected to server computer120. For example, I/O interface312may provide a connection to external devices318such as a keyboard, keypad, a touch screen, and/or some other suitable input device. External devices318can also include portable computer readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present invention, e.g., program200, can be stored on such portable computer readable storage media and can be loaded onto persistent storage308via I/O interface(s)312. I/O interface(s)312also connect to a display320.