Patent ID: 12212724

DESCRIPTION OF THE EMBODIMENTS

Hereafter, embodiments according to the present invention will be described in detail by referring to the drawings. However, components mentioned in the embodiment are mere examples, and these do not restrict the scope of the present invention.

FIG.1is a block diagram showing an information processing system according to the embodiment. As shown inFIG.1, the information processing system consists of a voice control apparatus100, an image forming apparatus (an image processing apparatus)101, a server (an information processing apparatus)102, a client terminal103, and a gateway105. It should be noted that the information processing system may be provided with a plurality of voice control apparatuses100, image forming apparatuses101, and client terminals103, respectively.

The voice control apparatus100, image forming apparatus101, and client terminal103are mutually communicable through the gateway105and a network104. Moreover, the voice control apparatus100, image forming apparatus101, and client terminal103are communicable with the server102through the gateway105and the Internet107.

In response to a voice operation start instruction from the user106, the voice control apparatus100obtains voice of a user106, codes the obtained voice (voice information of natural language), and transmits the coded voice data to the server102. Moreover, the voice control apparatus100outputs voice data received from the server102as voice. The voice control apparatus100is a voice input/output apparatus, such as a smart speaker or a smart phone, that enables voice communication with a user. In the embodiment, the voice control apparatus100and the image forming apparatus101are independent. However, the embodiment is not limited to this configuration. For example, hardware functions (hardware blocks mentioned later by referring toFIG.3) and software functions (software blocks mentioned later by referring toFIG.6) that constitute the voice control apparatus100may be included in the image forming apparatus101.

The image forming apparatus101is a multifunction apparatus equipped with a plurality of functions, such as a copy function, scan function, print function, and facsimile function, in this embodiment. However, the image forming apparatus101may be a printer or a scanner equipped with a single function. The following description presumes that the image forming apparatus101is a color laser beam multifunction apparatus. Moreover, the image forming apparatus101is provided with an operation panel108. The operation panel108is a display unit that displays a screen through which an operation instruction is received from the user106and that displays a state of the image forming apparatus101. Moreover, the operation panel108is provided with a touch panel (a display device)200that is united with an LCD display as shown inFIG.2mentioned later and functions also as an input unit that receives an operation of the user106.

The server102performs voice recognition of voice data of the user106obtained from the voice control apparatus100and determines words related to a setting operation of the image forming apparatus101or execution of a job on the basis of the result of the voice recognition. It should be noted that a job shows a unit of a series of image forming processes (for example, a copy process, scan process, print process, etc.) that the image forming apparatus101achieves using a print engine513and a scanner515inFIG.5mentioned later. Moreover, the server102generates a text depending on the result of the voice recognition or the above-mentioned word determination result and combines voice data that the voice control apparatus100reproduces contents of the text as voice. It should be noted that the server102can provide a high-precision voice recognition result by using machine learning, such as deep learning, with a neural network. For example, the server102learns so as to recognize a voice from a distant user correctly. Moreover, the server102is compatible with a natural language process and can obtain suitable information (a word, a kana-kanji conversion result in a case of Japanese) from input natural language by passing through a morphological analysis, syntax analysis, semantic analysis, context analysis, etc.

The client terminal103is a personal computer (PC) that the user106uses, for example. The client terminal103generates a print job for controlling the image forming apparatus101to print an electronic file. The electronic file is stored in an external storage unit505inFIG.5mentioned later, the client terminal103, or the like. Moreover, the client terminal103receives the image data that the image forming apparatus101scans and generates from the image forming apparatus101.

The network104mutually connects the voice control apparatus100, the image forming apparatus101, the client terminal103, and the gateway105. Various data, such as a print job and a scan job, and voice data are transmitted and received through the network104.

The gateway105is a wireless LAN router based on an IEEE802.11 standard series, etc., for example. The IEEE802.11 standard series includes a series of standards belonging to the IEEE802.11, such as IEEE802.11a and IEEE802.11b. It should be noted that the gateway105may have ability to operate according to other wireless communication systems different from the IEEE802.11 standard series. Moreover, the gateway105may not be a wireless LAN router, but may be a wired LAN router based on the Ethernet standard represented by 10BASE-T, 100BASE-T, 1000BASE-T, etc. and may have ability to operate according to another wired communication system.

FIG.2is a view showing an example of a home screen201displayed on the touch panel200of the operation panel108shown inFIG.1. The home screen201is displayed when the image forming apparatus101starts. UI widgets (touch objects) of functions that the image forming apparatus101executes are displayed on the home screen201. In the description, a UI widget means a certain area (a button, an icon, a mark, an arrow, a tab, and a rectangle) that the user106can distinguish on the touch panel200. When the user106touches a UI widget, a function associated with the touched UI widget is executed.

A copy button202is a UI widget that causes screen transition to a copy screen1112shown inFIG.11Dmentioned later on which setting required for execution of a copy function is performed. A scan button203is a UI widget that causes screen transition to a scan screen (not shown) on which setting required for execution of a scan function is performed. A menu button204is a UI widget that causes screen transition to a menu screen (not shown) on which a display language of the operation panel108is set. An address book button205is a UI widget that causes screen transition to an address book screen on which a transmission destination of image data that the image forming apparatus101scans and generates is set. A secure print button206is a UI widget that causes screen transition to a print screen (not shown) for printing image data equipped with a password that the image forming apparatus101receives. A voice recognition button207is a UI widget for enabling a voice operation by the voice control apparatus100. When the voice recognition button207is pressed and the voice operation by the voice control apparatus100becomes available, a message “Recognizing Voice” is displayed in a status display area210.

A state check button208is a UI widget for displaying information about a job that the image forming apparatus101executed and an active job. Tab numbers “1” through “7” are displayed in a tab region209. The home screen201consists of a plurality of page screens. When one of the tab numbers “2” through “7” is pressed, the screen is switched to a page screen corresponding to the pressed tab number. UI widgets different from the UI widgets shown inFIG.2are displayed on these page screens. Such a configuration enables a user to easily display the other UI widgets that are not displayed on the home screen201by easy press operation of a tab number. As mentioned above, the message “Recognizing Voice”, which shows that voice operation is available, is displayed in the status display area210. Moreover, job states, such as “Printing”, “Receiving”, “Transmitting”, and “Reading”, and errors, such as “Out of Sheet”, “Jam”, and “Out of Toner”, can be displayed in the status display area210.

FIG.3is a block diagram schematically showing a hardware configuration of the voice control apparatus100inFIG.1. As shown inFIG.3, the voice control apparatus100is provided with a controller unit300, a microphone308, a speaker (voice output device)310, and an LED312. The controller unit300is connected with the microphone308, speaker310, and LED312. Moreover, the controller unit300is provided with a CPU302, a RAM303, a ROM304, an external storage unit305, a network I/F (interface)306, a microphone I/F307, an audio controller309, and a display controller311. The CPU302, RAM303, ROM304, external storage unit305, network I/F306, microphone I/F307, audio controller309, and display controller311are mutually connected through a system bus301.

The CPU302is a central processing unit that controls entire operations of the controller unit103. The RAM303is a volatile memory. The ROM304is a nonvolatile memory and stores a boot program of the CPU302. The external storage unit305is a large capacity storage device (for example, an SD card) in comparison with the RAM303. It should be noted that the external storage unit305may be a storage device like a flash ROM other than an SD card as long as the storage device has a function equivalent to an SD card. The external storage unit305stores a control program for the voice control apparatus100executed by the controller unit300. For example, a voice control program601inFIG.6mentioned later is stored.

The CPU302runs the boot program stored in the ROM304when activating by turning a power source ON. The boot program is used to read a control program stored in the external storage unit305and to develop the control program onto the RAM303. When running the boot program, the CPU302continuously runs the control program developed onto the RAM303and controls voice input/output, display, and a data communication with the network104. Moreover, the CPU302stores data used during execution of the control program on the RAM303and reads and writes the data. The external storage unit305can store various setting values required for execution of the control program. The various setting values include a URL of the server102that enables access to the image forming apparatus101, for example and are read and written by the CPU302. The CPU302communicates with another apparatus on the network104through the network I/F306.

The network I/F306includes a circuit and an antenna for communication according to the wireless communication system based on the IEEE802.11 standard series. It should be noted that the communication system of the network I/F306is not limited to the wireless communication system. For example, the wired communication system based on the Ethernet standard may be employed. The microphone308is connected to the microphone I/F307. The microphone I/F307obtains voice that the user106utters through the microphone308, codes the obtained voice to convert it into voice data, and stores the voice data concerned in the RAM303in accordance with an instruction from the CPU302.

The microphone308is a voice operation device that can obtain voice of the user106. For example, a compact MEMS microphone mounted in a smart phone etc. is employed. Moreover, it is preferable to arrange three or more microphones308in order to calculate an arrival direction of the voice that the user106utters. It should be noted that this embodiment is achievable by the single microphone308and the number of the microphones is not limited to three or more. The speaker310is connected to the audio controller309. The audio controller309converts voice data into an analog voice signal in accordance with an instruction from the CPU302and outputs voice through the speaker310.

The speaker310is a general-purpose device for reproducing voice. The speaker310reproduces response voice showing that the voice control apparatus100is responding and voice combined by the server102.

The LED312is connected to the display controller311. The display controller311controls lighting of LED312in accordance with an instruction from the CPU302. For example, the display controller311controls the lighting of the LED312for showing that the voice control apparatus100is obtaining voice of the user106correctly. For example, the LED312is a blue LED that is visible to the user106. The LED312is a general-purpose device. It should be noted that the LED312can be replaced with a display device that can display a character and a picture.

FIG.4is a block diagram schematically showing a hardware configuration of the server102inFIG.1. As shown inFIG.4, the server102is provided with controller units400aand400b. It should be noted that the controller units400aand400bhave the same configuration in the embodiment. Hereinafter, the configuration of the controller unit400awill be described as an example. The controller unit400aruns a voice recognition program701inFIG.7mentioned later. The controller unit400ais provided with a CPU402aconnected to a system bus401a, a RAM403a, a ROM404a, an external storage unit405a, and a network I/F406a.

The CPU402ais a central processing unit that controls entire operations of the controller unit400a. The RAM403ais a volatile memory. The ROM404ais a nonvolatile memory and stores a boot program of the CPU402a. The external storage unit405ais a large capacity storage unit (for example, a hard disk drive: HDD) in comparison with the RAM403a. The external storage unit405astores a control program for the server102executed by the controller unit400a. For example, the voice recognition program701inFIG.7mentioned later is stored. It should be noted that the external storage unit405amay be a storage device like a solid state drive (SSD) other than a hard disk drive as long as the storage device has a function equivalent to a hard disk drive. Moreover, the external storage unit405amay be an external storage device that the server102can access.

The CPU402aruns the boot program stored in the ROM404awhen activating by turning a power source ON. The boot program is used to read a control program stored in the external storage unit405aand to develop the control program onto the RAM403a. When running the boot program, the CPU402acontinuously runs the control program developed onto the RAM403aand controls. Moreover, the CPU402astores data used during execution of the control program on the RAM403aand reads and writes the data. The external storage unit405acan store various setting values required for execution of the control program. The various setting values are read and written by the CPU402a. The CPU402acommunicates with another apparatus on the network104through the network I/F406a.

The controller unit400bruns a remote control program801inFIG.8mentioned later. The controller unit400bis provided with a CPU402bconnected to the system bus401b, a RAM403b, a ROM404b, an external storage unit405b, and a network I/F406b. The CPU402b, RAM403b, ROM404b, external storage unit405b, and network I/F406brespectively have the same functions and configurations with the CPU402a, RAM403a, ROM404a, external storage unit405a, and network I/F406a. The external storage unit405bstores a control program for the server102executed by the controller unit400b. For example, the remote control program801inFIG.8mentioned later is stored. Although the configuration in which the different controller units respectively execute the voice recognition program701and the remote control program801that are mentioned later is described in the embodiment, the present invention is not limited to this configuration. For example, one of the controller units400aand400bmay run both of the voice recognition program701and the remote control program801that are mentioned later.

FIG.5is a block diagram schematically showing a hardware configuration of the image forming apparatus101inFIG.1. As shown inFIG.5, the image forming apparatus101is provided with a controller unit500, the operation panel108, the print engine (printing device)513, and the scanner (scanning device)515. The controller unit500is connected with the operation panel108, the print engine513, and the scanner515. Moreover, the controller unit500is provided with a CPU502, a RAM503, a ROM504, an external storage unit505, a network I/F506, a display controller507, an operation I/F508, a print controller512, and a scan controller514. The CPU502, RAM503, ROM504, external storage unit505, network I/F506, display controller507, operation I/F508, print controller512, and scan controller514are mutually connected through a system bus501.

The CPU502is a central processing unit that controls entire operations of the controller unit500. The RAM503is a volatile memory. The ROM504is a nonvolatile memory and stores a boot program of the CPU502. The external storage unit505is a large capacity storage unit (for example, a hard disk drive: HDD) in comparison with the RAM503. The external storage unit505stores a control program for the image forming apparatus101executed by the controller unit500. For example, a device control program901inFIG.9mentioned later is stored. It should be noted that the external storage unit505may be a storage device like a solid state drive (SSD) other than a hard disk drive as long as the storage device has a function equivalent to a hard disk drive.

The CPU502runs the boot program stored in the ROM404awhen activating by turning a power source ON. The boot program is used to read a control program stored in the external storage unit505and to develop the control program onto the RAM503. When running the boot program, the CPU502continuously runs the control program developed onto the RAM503and controls. Moreover, the CPU502stores data used during execution of the control program on the RAM503and reads and writes the data. The external storage unit505can further store various setting values required for execution of the control program and image data read by the scanner515. The CPU502reads and writes the setting values and the image data. The CPU502communicates with another apparatus on the network104through the network I/F506and the server102on the Internet through the gateway105.

The operation panel108is connected to the display controller507and the operation I/F508. The display controller507controls screen display of the touch panel200of the operation panel108in accordance with an instruction from the CPU502. The operation I/F508outputs and inputs an operation signal. When the touch panel200is pressed, the CPU502obtains a coordinate that shows the pressed position on the touch panel200through the operation I/F508.

The print engine513is connected to the print controller512. The print controller512transmits control commands and image data to the print engine513in accordance with the instructions from the CPU502. The print engine513performs the print process that prints the received image data on sheets in accordance with a control command received from the print controller512. The scanner515is connected to the scanner controller514. The scan controller514transmits a control command to the scanner515in accordance with the instruction from the CPU502and writes the image data received from the scanner515into the RAM503. The scanner515performs a reading process that reads a document on a platen glass (not shown) of the image forming apparatus101with an optical unit in accordance with a control command received from the scan controller514.

FIG.6is a block diagram showing a functional configuration of the voice control program601that the CPU302of the voice control apparatus100inFIG.3runs. The voice control program601is stored in the external storage unit305as mentioned above. The CPU302develops the voice control program601onto the RAM303on boot and runs it. The voice control program601consists of a data transmission/reception module602, a data management module603, a voice obtainment module604, a voice reproduction module605, a display module606, a voice-operation-start detection module607, an utterance end determination module608, and a voice control module609.

The data transmission/reception module602transmits and receives data with another apparatus on the network104by TCP/IP through the network I/F306. The data transmission/reception module602transmits the voice data obtained from the voice obtainment module604to the server102. Moreover, the data transmission/reception module602also receives combined voice data (a response to the user106) generated by the server102.

The data management module603stores various data like work data etc. that are generated during execution of the voice control program601in a predetermined area of the external storage unit305and manages it. For example, volume setting data of voice reproduced by the voice reproduction module605, authentication information required for communication with the gateway105, device information required for communication with the server102and the image forming apparatus101, a URL of a program are stored and managed.

The voice obtainment module604converts analog voice of the user106near the voice control apparatus100obtained with the microphone308into voice data and temporarily stores the voice data concerned in the RAM303. The voice of the user106is converted into a predetermined format like MP3, is coded to be transmitted to the server102, and is temporarily stored in the RAM303as the voice data. A start timing and an end timing of the process by the voice obtainment module604are managed by the voice control module609. It should be noted that the coding format may be a general-purpose streaming format and that the data transmission/reception module602may sequentially transmit the coded voice data.

The voice reproduction module605reproduces the combined voice data, which is received by the data transmission/reception module602, with the speaker310through the audio controller309. A voice reproduction timing of the voice reproduction module605is managed by the voice control module609. The display module606controls lighting of the LED312through the display controller311. For example, when the voice-operation-start detection module607detects a voice operation, the display module606lights the LED312. A lighting timing of the display module606is managed by the voice control module609.

When detecting utterance of a wake word by the user106, press of the operation start key (not shown) of the voice control apparatus100, or receipt of a voice control start command by the data transmission/reception module602, the voice-operation-start detection module607transmits an operation start notification to the voice control module609. In the description, the wake word is a predetermined voice word for starting a voice operation of the voice control apparatus100. The voice-operation-start detection module607always checks the wake word in the analog voice of the user106who is adjacent to the voice control apparatus100obtained with the microphone308. The user106can operate the image forming apparatus101by uttering the wake word and by uttering an operation instruction continuously.

The utterance end determination module608determines an end timing of the process by the voice obtainment module604. For example, when the voice of the user106breaks off in a predetermined period (for example, 3 seconds), the utterance end determination module608determines that the utterance of the user106is finished, and transmits an utterance end notification to the voice control module609. It should be noted that the utterance end may be determined by a predetermined word that the user106utters in place of a no-utterance period (hereinafter referred to as a blank period). For example, when a predetermined word, such as “yes”, “no”, “OK”, “cancel”, “finish”, or “start”, is detected, the utterance end may be determined without waiting for the predetermined period. Moreover, the utterance end may be determined by the server102not by the voice control apparatus100. The utterance end may be determined on the basis of meaning and context of contents of utterance of the user106.

The voice control module609is the center of control and controls so that other modules in the voice control program601cooperate and operate mutually. Specifically, the voice control module609controls starts and ends of processes of the voice obtainment module604, voice reproduction module605, and display module606. Moreover, the voice control module609controls so that the data transmission/reception module602transmits the voice data concerned to the server102after the voice obtainment module604converts the voice obtained with the microphone308into the voice data. Moreover, the voice control module609controls so that the voice reproduction module605reproduces the combined voice data after the data transmission/reception module602receives the combined voice data from the server102.

The timings of starts and ends of the processes of the voice obtainment module604, voice reproduction module605, and display module606will be described.

The voice control module609starts the process of the voice obtainment module604when receiving the operation start notification from the voice-operation-start detection module607. Moreover, the voice control module609finishes the process of the voice obtainment module604when receiving the utterance end notification from the utterance end determination module608. For example, the user106shall utter the wake word and continuously shall utter “start copy”. At this time, the voice-operation-start detection module607detects the voice of the wake word and transmits the operation start notification to the voice control module609. The voice control module609controls to start the process of the voice obtainment module604, when receiving the operation start notification. The voice obtainment module604converts the analog voice “start copy” uttered following the wake word into the voice data and stores the voice data concerned temporarily. When determining that the blank period after the utterance “start copy” is longer than the predetermined period, the utterance end determination module608transmits the utterance end notification to the voice control module609. When receiving the utterance end notification, the voice control module609finishes the process of the voice obtainment module604. It should be noted that the state between the start and end of the process by the voice obtainment module604is called an utterance processing state. The display module606lights the LED312in the utterance processing state.

After the utterance end determination of the user106, the voice control module609controls the data transmission/reception module602to transmit the voice data to the server102and waits for a response from the server102. For example, the response from the server102consists of a header section showing that it is a response and combined voice data for reproducing a response message. When the data transmission/reception module602receives the response from the server102, the voice control module609controls the voice reproduction module605to reproduce the combined voice data included in the response from the server102. The combined voice data reproduces a message “copy screen is displayed”, for example. It should be noted that the state after the utterance end determination until finishing reproduction of the combined voice data shall be called a response processing state. The display module606blinks the LED312in the response processing state.

After the response process, while the interactive session with the server102is continuing, the user106can continuously utter words to perform a desired process without uttering the wake word. The interactive session is finished when the server102transmits an interactive session end notification to the voice control apparatus100. It should be noted that the state from the end of the interactive session to the start of the following interactive session shall be called a standby state. The voice control apparatus100will be in the standby state until the voice control apparatus100receives the operation start notification from the voice-operation-start detection module607. The display module606turns off the LED312in the standby state.

FIG.7is a block diagram showing a functional configuration of the voice recognition program701that the CPU402aof the server102inFIG.4runs. The voice recognition program701is stored in the external storage unit405aas mentioned above. The CPU402adevelops the voice recognition program701onto the RAM403aon boot and runs it. The voice recognition program701consists of a data transmission/reception module702, a data management module703, and a voice data conversion module704.

The data transmission/reception module702transmits and receives data with another apparatus on the network104by TCP/IP through the network I/F406a. The data transmission/reception module702receives the voice data of the user106from the voice control apparatus100.

The data management module703stores various data, such as work data generated in execution of the voice recognition program701and parameters that are required for execution of the voice recognition process by the voice data conversion module704, in the predetermined area of the external storage unit405aand manages the data. For example, the data management module703stores an acoustic model and a language model that are used to convert the voice data received by the data transmission/reception module702into a text (text information) in the predetermined area of the external storage unit405aand manages them. Moreover, the data management module703stores a dictionary that is used to perform the morphological analysis of a text by a morphological analysis module706mentioned later in the predetermined area of the external storage unit405aand manages it. Moreover, the data management module703stores a voice database that is used to combine voices by a voice combination module707mentioned later in the predetermined area of the external storage unit405aand manages it. Furthermore, the data management module703stores device information required for communications with the voice control apparatus100and image forming apparatus101and manages it.

The voice data conversion module704consists of a voice recognition module705, the morphological analysis module706, and the voice combination module707. The voice recognition module705performs the voice recognition process for converting the voice data of the user106received by the data transmission/reception module702into a text. The voice recognition process converts the voice data of the user106into phonemes using the acoustic model and converts the phonemes into the data of actual text format by pattern matching using the language model. It should be noted that the acoustic model may be a model that uses the machine learning technique by a neural network like DNN-HMM or may be a model that uses a different technique like GMM-HMM. In the machine learning using the neural network, the learning model is learned on the basis of teacher data that pairs voice and text, for example. The language model may be a model that uses the machine learning technique by the neural network like RNN or may be a model that uses a different technique like the N-gram method.

In the embodiment, the data of the above-mentioned text format shall consist of a text that consists of one or more characters (a numerical character, an alphabet, a symbol, etc. may be included). In the case of Japanese, the data shall consist of a text to which kana-kanji conversion has been applied. However, another method may be used as the voice recognition process that converts voice data into data of the text format and a method is not restricted to the method mentioned above.

The morphological analysis module706applies the morphological analysis to the data of the text format converted by the voice recognition module705. The morphological analysis deduces a morpheme string from the dictionary, which has the grammar of the language and information about word classes, and determines the word class of each morpheme. The morphological analysis module706can be achieved using the well-known morphological analysis software, such as JUMAN, ChaSen, and MeCab in the case of Japanese, and Tree Tagger in the case of English. For example, the morphological analysis module706analyzes the data “kopiwoshitai (I want to copy)” in the Japanese text format converted by the voice recognition module705as the morpheme string “kopi”, “wo”, “shi”, and “tai”. Moreover, the data “A3karaA4he (from A3 to A4)” is analyzed as the morpheme string “A3”, “kara”, “A4”, and “he”.

The voice combination module707performs a voice combination process on the basis of the notification received from the image forming apparatus101. The voice combination process converts a predetermined text that is paired to a predetermined notification into combined voice data in a predetermined format, such as MP3. A voice combination process generates combined voice data on the basis of the voice database stored in the data management module703, for example. The voice database collects voices that are utterances of contents of fixed forms, such as words, for example. Although the voice combination process is performed using the voice database in the embodiment, the voice combination process may be performed by another method. The method is not limited to the method using the voice database.

FIG.8Ais a block diagram showing a functional configuration of a remote control program801that the CPU402bof the server102inFIG.4runs. The remote control program801is stored in the external storage unit405bas mentioned above. The CPU402bdevelops the remote control program801onto the RAM403bon boot and runs it. The remote control program801consists of a data transmission/reception module802, a data management module803, and a remote control data conversion module804.

The data transmission/reception module802transmits and receives data with another apparatus on the network104by TCP/IP through the network I/F406b. Moreover, the data transmission/reception module802receives text data (word information) obtained from the voice recognition program701by performing the morphological analysis. This text data is the data in the text format that consists of one or more morphemes obtained by the voice recognition program701performing the morphological analysis to the voice data of the user106.

The data management module803stores various data like parameters required for the remote control of the image forming apparatus101based on the voice of the user106obtained by the voice control apparatus100in the predetermined area of the external storage unit405band manages the data. Specifically, the data management module803stores and manages screen information, UI widget information, a voice recognition message, an operation instruction, and a filter word that are shown inFIG.8B.

The screen information indicates a type of a screen, such as a home screen and a copy screen, displayed on the touch panel200of the image forming apparatus101. It should be noted that the data management module803manages the screen information about all page screens of a screen like the home screen that has a plurality of page screens that are switchable by tabs in the embodiment.

The UI widget information indicates a name of a UI widget displayed on a screen shown by a screen type. For example, the names of the UI widgets, such as the copy button202, scan button203, menu button204, and the tub numbers, shown inFIG.2are managed as the UI widget information. Although the embodiment describes the case where the name of the UI widget is managed as the UI widget information, a feature other than the name may be managed as long as the feature indicates the information about the UI widget. For example, the UI widget may be managed by coordinate information that shows a displayed position of the UI widget on the screen or managed by a display form (a button, an icon, an input box, or the like) of the UI widget.

The voice recognition message indicates a phrase for recognizing the UI widget shown by the UI widget information by voice. Moreover, the voice recognition message includes at least one word related to the process executed by the function corresponding to the UI widget concerned and has a semantic relation to the executed process concerned. The different voice recognition messages are set to the respective UI widgets. It should be noted that there is no voice recognition message about the secure print206that does not allow the voice operation from a viewpoint of security. This is denoted by “-” inFIG.8B. The remote control program801specifies the UI widget that is subjected to the voice operation on the basis of the UI widget information and the voice recognition message that are managed by the data management module803.

An operation instruction is transmitted to the image forming apparatus101from the server102, when the text data received from the voice recognition program701matches the voice recognition message. The filter word is used to perform filter display of one or more UI widgets displayed on the touch panel200. During the filter display, one or more target UI widgets are extracted by filtering and the extracted target UI widgets are displayed on the touch panel200. A unique filter word is set to each function group to which UI widgets belong. For example, the filter word “copy” is set to the group of the UI widgets relevant to the copy function. And the filter word “scan” is set to the group of the UI widgets relevant to the scan function. If a filter word matches a voice recognition message, filter display becomes impossible. Accordingly, a filter word consists of a word different from a voice recognition message, such as a part of words constituting the voice recognition message and a different word that has meaning similar to the voice recognition message.

The remote-control-data conversion module804consists of a control command analysis module805, voice/character conversion module806, and screen configuration obtainment module807. The control command analysis module805compares the text data received from the voice recognition program701with the voice recognition messages that the data management module803manages. When the managed voice recognition messages include the voice recognition message that matches the text data, the remote-control-data conversion module804transmits the operation instruction corresponding to this voice recognition message to the image forming apparatus101. In the meantime, when the managed voice recognition messages are not coincident with the text data, the remote-control-data conversion module804compares the text data with the filter words that the data management module803manages. When the managed filter words include the filter word that matches the text data, the remote-control-data conversion module804transmits the voice recognition message corresponding to this filter word and the UI widget information corresponding to the voice recognition message to the image forming apparatus101.

The voice/character conversion module806transmits the text information about the voice data output from the speaker310of the voice control apparatus100to the voice recognition program701in response to reception of a screen update notification mentioned later from the image forming apparatus101.

The screen configuration obtainment module807requests the data management module803to store the screen information and UI widget information received from the image forming apparatus101after transmitting the operation instruction. Moreover, the screen configuration obtainment module807requests the data transmission/reception module802to transmit the voice recognition messages that will be displayed in association with the UI widget included on the screen displayed on the touch panel200of the image forming apparatus101to the image forming apparatus101.

FIG.9is a block diagram showing a functional configuration of a device control program901that the CPU502of the image forming apparatus101inFIG.5runs. The device control program901is stored in the external storage unit505as mentioned above. The CPU502develops the device control program901onto the RAM503on boot and runs it. The device control program901consists of a data transmission/reception module902, a data management module903, a scan module904, a print unit905, a display unit906, a voice operation determination module907, and a device control module908.

The data transmission/reception module902transmits and receives data with another apparatus on the network104by TCP/IP through the network I/F506. For example, the data transmission/reception module902receives the operation instruction and the filter display instruction from the server102. Moreover, the data transmission/reception module902transmits the screen update notification showing that the screen display contents of the touch panel200have been updated and the job execution state notification that shows the state of the job to the server102.

The data management module903stores the various data, such as the work data generated by execution of the device control program901and the setting parameters required for controlling the devices, in the predetermined areas of the RAM503and external storage unit505and manages the data. For example, the setting items of the job executed by the device control module908, the job data constituted by the combination of the set values, and the attribute information about sheets are managed. Moreover, the authentication information required for communication with the gateway105, the device information required for communication with the server102, and the URL are managed. Moreover, the image data that is subjected to the image formation by the image forming apparatus101is stored and managed. Moreover, the screen control information that the display module906uses for screen display control and the voice operation determination information that the voice operation determination module907uses to determine an operation are stored. The screen control information and voice operation determination information are managed for each screen that the display module906displays. Moreover, instructions and control means for booting the voice recognition and voice operation through the network I/F506or another boot means are managed.

The scan module904executes the scan with the scanner515through the scan controller514on the basis of the scan job parameter setting of the device control module908and stores the read image data with the management module903. The print module905executes the print with the print engine513through the print controller512on the basis of the print job parameter setting of the device control module908.

The display module906controls the operation panel108through the display controller507. The display module906displays the UI widgets (buttons, pulldown lists, check boxes, etc.) that the user can operate on the touch panel200on the basis of the above-mentioned screen control information. Moreover, the display module906obtains a coordinate that indicates a touched position on the touch panel200through the operation I/F508. Then, the display module906determines the UI widget that is subjected to the operation and also determines contents of a process at the time of receiving the operation. The display module906updates the contents of the screen displayed on the touch panel200according to the determined contents of the process or transmits the parameters of the job set up by the user's operation and a start instruction of the job concerned to the device control module908. Moreover, when the voice operation is started, as shown inFIG.11Amentioned later, the display module906displays voice recognition messages in association with the UI widgets. Then, the display module906updates the contents of the screen displayed on the touch panel200according to the voice operation determination result of the voice operation determination module907or transmits the parameters of the job set up by the user's operation and a start instruction of the job concerned to the device control module908.

The voice operation determination module907determines the UI widget that constitutes the screen displayed on the touch panel200and is operable by the user as the operation target on the basis of the operation instruction that the data transmission/reception module902received from the server102. For example, when the operation instruction “start copy” is received in the state where the home screen201is displayed, the copy screen1112inFIG.11Dmentioned later is displayed on the touch panel200of the image forming apparatus101. When the operation instruction “start” is received in this state, the image forming apparatus101executes a copy process. In this way, after displaying the status under the voice recognition, the image forming apparatus101starts the copy process in a default setting state in the copy screen, when the user106utters “copy” and “start” to the voice control apparatus100.

The device control module908instructs control of the print engine513and the scanner515through the print controller512and the scan controller514. For example, when press of a start key of the touch panel200is detected in the state where the copy screen1112inFIG.11Dmentioned later is displayed on the touch panel200, the device control module908receives the parameters of the copy job and the job start instruction from the display module906. The device control module908controls the print engine513so as to print the image data read with the scanner515onto a sheet on the basis of the job parameters.

FIG.10is a sequence chart showing procedures of a voice operation control process executed by the information processing system according to the embodiment. In the embodiment, the CPU302of the voice control apparatus100develops the voice control program601stored in the external storage unit305onto the RAM303and runs it. Moreover, the CPU402aof the server102develops the voice recognition program701stored in the external storage unit405aonto the RAM403aand runs it. The CPU402bdevelops the remote control program801stored in the external storage unit405onto the RAM403band runs it. Furthermore, the CPU502of the image forming apparatus101develops the device control program901stored in the external storage unit505onto the RAM503and runs it. Thereby, the voice operation control process inFIG.10is achieved.FIG.10describes a case where the user106utters “start voice operation”, “copy”, and “start copy” as an example. The display control process inFIG.10assumes that the image forming apparatus101starts and the home screen201is displayed on the touch panel200.

InFIG.10, the user106utters “start voice operation” to the microphone308of the voice control apparatus100first (step S1001). The voice control program601stores “start voice operation” that the user106uttered into the external storage unit305as voice data. Next, the voice control program601transmits the voice data “start voice operation” to the server102(step S1002).

The voice recognition program701of the server102performs the voice recognition process based on the received voice data (step S1003). Specifically, the voice recognition program701stores the received voice data into the external storage unit405aand converts the voice data concerned into text data with the voice recognition module705. Next, the voice recognition program701transmits the converted text data to the remote control program801(step S1004).

The remote control program801specifies the operation instruction corresponding to the received text data (specifically, “start voice operation”) from among the operation instructions that the data management module803manages and transmits the specified instruction to the image forming apparatus101(step S1005).

When receiving the operation instruction “start voice operation”, the device control program901of the image forming apparatus101displays a message showing that a voice operation is available (specifically, “recognizing voice”) in the status display area210on the home screen201(step S1006). Next, the device control program901transmits screen information that shows the type of the screen displayed on the touch panel200and UI widget information that shows UI widgets included in the screen concerned to the server102(step S1007). Specifically, the device control program901transmits the screen information “home screen201” that shows the home screen201displayed on the touch panel200to the server102.

The remote control program801of the server102specifies one or more pieces of UI widget information corresponding to the home screen201on the basis of the screen information received from the image forming apparatus101from among the pieces of UI widget information that the data management module803manages. Moreover, the remote control program801specifies one or more voice recognition messages corresponding to the specified UI widget information from among the voice recognition messages that the data management module803manages. The remote control program801transmits all the pieces of specified UI widget information and all the specified voice recognition messages to the image forming apparatus101(step S1008).

The device control program901of the image forming apparatus101displays a voice recognition message as an utterance example in association with each UI widget displayed on the home screen201on the basis of the received UI widget information (step S1009). Specifically, the device control program901displays a balloon1101including “start copy”, which is a voice recognition message of the copy button202, as shown inFIG.11Ain association with the copy button202that is an example of a UI widget that allows the voice operation. Moreover, the device control program901displays a balloon1102showing that the voice operation is impossible in association with the secure print button206that is an example of a UI widget that prohibits the voice operation as shown inFIG.11A. Furthermore, the device control program901displays a message “operate with voice by uttering displayed word” in the status display area210and displays a balloon1103including “erase word” in association with the status display area210as shown inFIG.11A. The voice recognition message “erase word” is used to hide the voice recognition messages that are displayed on the screen of the touch panel200. Next, the device control program901transmits the screen update notification showing that the screen display contents of the touch panel200have been updated to the server102(step S1010).

When receiving the screen update notification, the remote control program801of the server102transmits response text data, which is a response message with respect to the content that the user106uttered in the step S1001, to the voice recognition program701(step S1011). Specifically, the remote control program801transmits the response text data including a message “operation is possible using displayed words” to the voice recognition program701. The voice recognition program701converts the received response text data into combined voice data and transmits the combined voice data concerned to the voice control apparatus100(step S1012).

The voice control program (a voice output control unit)601of the voice control apparatus100outputs the received combined voice data from the speaker310. Specifically, the voice control program601outputs a voice message “operation is possible using displayed words” from the speaker310(step S1013). In this way, the voice operation of the image forming apparatus101becomes available when the user106utters “start voice operation” to the microphone308in the embodiment.

After that, when the user106utters “copy” to the microphone308of the voice control apparatus100(step S1014), the voice control program601stores voice data “copy” that the user106uttered into the external storage unit305. Next, the voice control program601transmits the voice data “copy” to the server102(step S1015).

The voice recognition program701of the server102performs the voice recognition process based on the received voice data (step S1016) as with the step1003. Next, the voice recognition program701transmits the converted text data to the remote control program801(step S1017). Next, the remote control program801compares the received text data with the information which the data management module803manages. Specifically, the remote control program801compares the received text data with the voice recognition messages that the data management module803manages. For example, the voice recognition messages inFIG.8Bthat the data management module803manages do not include a voice recognition message that matches the received text data “copy”. In this case, the remote control program801compares the received text data with the filter words that the data management module803manages. For example, the filter words inFIG.8Bthat the data management module803manages include a filter word that matches “copy”. In this case, the remote control program801specifies one or more pieces of UI widget information of which the corresponding filter word is “copy” from among the pieces of UI widget information that the data management module803manages. Moreover, the remote control program801specifies one or more voice recognition messages corresponding to the specified UI widget information from among the voice recognition messages that the data management module803manages. The remote control program801transmits a filter display instruction that includes all the pieces of specified UI widget information and all the specified voice recognition messages to the image forming apparatus101(step S1018).

The device control program901of the image forming apparatus101extracts the UI widgets relevant to “copy” by filtering and displays the extracted UI widgets according to the received filter display instruction (step S1019). Thereby, a home screen1104shown inFIG.11Bis displayed on the touch panel200. The home screen1104includes UI widgets relevant to “copy”, such as the copy button202, a show-through prevention copy button1105, a toner save copy button1106, a preset ID card copy button1107, a thickening character copy button1108, a passport copy button1109, and an ID card copy button1110. Moreover, the voice recognition messages are additionally displayed as utterance examples in association with these UI widgets as with the above-mentioned step S1009. Furthermore, the message “copy related widgets are extracted by filtering and displayed” is displayed in the status display area210. In the step S1019, the device control program901may display a list that collects the functions relevant to “copy” like a balloon1111inFIG.11C. In this way, when the functions relevant to “copy” are displayed as the list, a user can find a desired function easily, and operability of a voice operation is improved. Moreover, when all the UI widgets relevant to “copy” cannot be displayed on the home screen, the UI widgets relevant to “copy” are collectively displayed on a popup screen so as to be scrolled in the embodiment. When the number of the UI widgets relevant to “copy” is more than the predetermined number set up beforehand, the voice control apparatus100may be controlled so as to output a voice message “matched items are too many”. Next, the device control program901transmits the screen update notification showing that the screen display contents of the touch panel200have been updated to the server102(step S1020). The screen update notification includes the screen information showing the type of the screen displayed on the touch panel200in the step S1019.

When receiving the screen update notification, the remote control program801of the server102stores the screen information included in the screen update notification into the external storage unit405b. Moreover, the remote control program801transmits the response text data including the message “filtered by using copy” to the voice recognition program701(step S1021). The voice recognition program701converts the received response text data into combined voice data and transmits the combined voice data concerned to the voice control apparatus100(step S1022).

The voice control program601of the voice control apparatus100outputs the received combined voice data from the speaker310. Specifically, the voice control program601outputs a voice message “filtered by using copy” from the speaker310(step S1023). In this way, when the user106utters “copy” to the microphone308, the UI widgets relevant to “copy” are extracted by filtering and displayed on the touch panel200of the image forming apparatus101in the embodiment.

After that, the user106utters “start copy” to the microphone308of the voice control apparatus100(step S1024). The voice control program601stores “start copy” that the user106uttered into the external storage unit305as voice data. Next, the voice control program601transmits the voice data “start copy” to the server102(step S1025).

The voice recognition program701of the server102executes the voice recognition process (step S1026) on the basis of the received voice data as with the step S1003mentioned above and transmits the converted text data to the remote control program801(step S1027).

The remote control program801specifies the operation instruction corresponding to the received text data (specifically, “start copy”) from among the operation instructions that the data management module803manages and transmits the specified instruction to the image forming apparatus101(step S1028).

When receiving the operation instruction (“start copy”), the device control program901of the image forming apparatus101displays the copy screen for executing a copy job on the touch panel200(step S1029). Furthermore, at this time point, voice recognition messages like that shown inFIG.11Aare not displayed on the copy screen. Next, the device control program901transmits screen information that shows the type of the screen displayed on the touch panel200and UI widget information that shows UI widgets included in the screen concerned to the server102(step S1030).

The remote control program801of the server102specifies one or more pieces of UI widget information corresponding to the copy screen on the basis of the received screen information from among the pieces of UI widget information that the data management module803manages. Moreover, the remote control program801specifies one or more voice recognition messages corresponding to the specified UI widget information from among the voice recognition messages that the data management module803manages. The remote control program801transmits all the pieces of specified UI widget information and all the specified voice recognition messages to the image forming apparatus101(step S1031).

The device control program901of the image forming apparatus101displays a voice recognition message as an utterance example in association with each UI widget displayed on the copy screen201on the basis of the received UI widget information (step S1032). In the step S1032, balloons including voice recognition messages corresponding to the respective UI widgets included in the copy screen1112shown inFIG.11Dare displayed in association with the UI widgets as utterance examples. Next, the device control program901transmits the screen update notification including the screen information that shows the type of the screen displayed on the touch panel200to the server102(step S1033).

When receiving the screen update notification, the remote control program801of the server102transmits response text data, which is a response message with respect to the content that the user106uttered in the step S1024, to the voice recognition program701(step S1034). Specifically, the remote control program801transmits the response text data including a message “copy screen is displayed” to the voice recognition program701. The voice recognition program701converts the received response text data into combined voice data and transmits the combined voice data concerned to the voice control apparatus100(step S1035).

The voice control program601of the voice control apparatus100outputs the received combined voice data from the speaker310. Specifically, the voice control program601outputs a voice message “copy screen is displayed” from the speaker310(step S1036).

As mentioned above, the voice operation of the image forming apparatus101is performed on the basis of the voice of the user that the voice control apparatus100obtains in the embodiment.

Although the processes inFIG.10show the case where the voice recognition messages are displayed in association with the UI widgets, the voice recognition messages can be hidden.

For example, when the user106utters “erase words” to the microphone308in the state where the voice recognition messages are displayed in association with the UI widgets included in the home screen201as shown inFIG.11A, the voice control apparatus100transmits voice data “erase words” to the server102.

The server102performs the voice recognition process on the basis of the received voice data to generate the text data. The server102specifies the operation instruction corresponding to the above-mentioned text data from among the operation instructions that the data management module803manages and transmits the specified operation instruction (specifically, “erase words”) to the image forming apparatus101.

When receiving the operation instruction (“erase words”), the image forming apparatus101hides the voice recognition messages on the home screen201. Thereby, the home screen201(seeFIG.2) that includes no voice recognition message is displayed on the touch panel200. Next, the image forming apparatus101transmits the screen update notification including the screen information that shows the type of the screen displayed on the touch panel200to the server102.

When receiving the screen update notification, the server102converts the response text data showing the response message with respect to the content uttered by the user106into the combined voice data and transmits the combined voice data concerned to the voice control apparatus100.

The voice control apparatus100outputs the received combined voice data from the speaker310. Specifically, the voice control program601outputs the voice message “words are hidden and voice operation is available” from the speaker310.

In this way, display and non-display of the voice recognition messages can be easily switched on the basis of the voice that the user uttered in the embodiment.

Next, operations of the image forming apparatus101, server102, and voice control apparatus100for achieving the voice operation of the information processing system in the embodiment will be described.

FIG.12is a flowchart showing procedures of a screen update control process executed by the device control program901inFIG.9. The screen update control process inFIG.12is achieved because the CPU502of the image forming apparatus101develops the device control program901stored in the external storage unit505onto the RAM503and runs it. The screen update control process inFIG.12assumes that the image forming apparatus101starts and the home screen201is displayed on the touch panel200.

As shown inFIG.12, when receiving voice operation information, such as the operation instruction or the filter display instruction, from the server102(step S1201), the device control program901determines whether the received operation information is an operation instruction (step S1202).

As a result of the determination in the step S1202, when the received voice operation information is the operation instruction, the device control program901controls the display of the touch panel in accordance with the operation instruction (step S1203). For example, when the operation instruction is “start voice operation”, the device control program901display “recognizing voice” on the status display area210of the home screen201(see the step S1006mentioned above). Moreover, when the operation instruction is “display menu”, the device control program901displays a menu screen (not shown) on the touch panel200. Moreover, when the operation instruction is a job execution instruction, the device control program901executes the job concerned in parallel. For example, when the operation instruction is an operation instruction of a start button on the copy screen1112, a copy job is executed and a copy execution screen (not shown) showing that the copy job concerned is under execution is displayed on the touch panel200.

Next, the device control program901transmits screen information that shows the type of the screen displayed on the touch panel200and UI widget information that shows UI widgets included in the screen concerned to the server102(step S1204). When receiving the screen information from the image forming apparatus101, the server102specifies one or more pieces of UI widget information corresponding to the screen information concerned from among the pieces of UI widget information that the data management module803manages. Moreover, the server102specifies one or more voice recognition messages corresponding to the specified UI widget information from among the voice recognition messages that the data management module803manages. The remote control program801transmits all the pieces of specified UI widget information and all the specified voice recognition messages to the image forming apparatus101.

Next, the device control program901receives the UI widget information and the voice recognition messages from the server102(step S1205). Next, the device control program901displays the voice recognition messages in association with the UI widgets displayed on the touch panel200(step S1206). See the steps S1009and S1032mentioned above, for example. At this time, if the UI widgets corresponding to the voice recognition messages received from the server102are not displayed on the touch panel200, the device control program901may collectively display the voice recognition messages of which the corresponding UI widgets are not displayed as a list like the balloon1111inFIG.11C. Next, the device control program901transmits the screen update notification including the screen information that shows the type of the screen displayed on the touch panel200to the server102(step S1207) and finishes the process.

As a result of the determination in the step S1202, when the received voice operation information is not the operation instruction, the device control program901determines whether the received voice operation information is the filter display instruction (step S1208).

As a result of the determination in the step S1208, when the received voice operation instruction is the filter display instruction, the device control program901performs the filter display according to the received filter display instruction (step S1209). See the step S1019mentioned above, for example. As a result of the process in the step S1209, the screen that includes the UI widgets filtered using the word that the user106utters to the microphone308on the basis of the UI widget information included in the received filter display instruction is displayed on the touch panel200. Moreover, the voice recognition messages included in the received filter display instruction are displayed in association with the respective UI widgets. After that, the screen update control process proceeds to the step S1207. At this time, the screen update notification transmitted in the step S1207may include the screen information about the previous screen that was displayed on the touch panel200immediately before performing the filter display. Moreover, screen information about a new screen different from the previous screen may be included.

As a result of the determination in the step S1208, when the received voice operation information is not the filter display instruction, the screen update control process proceeds to a step S1210. For example, when the voice operation information shows the operation instruction of the secure print206that does not allow the voice operation, the received voice operation information is not the filter display instruction. In the step S1210, the device control program901transmits the voice operation failure response, which shows that the impracticable instruction is received, to the server102. After that, the screen update control process is finished. After finishing the screen update control process, the device control program901comes into a reception waiting state for the voice operation information from the server102.

FIG.13is a flowchart showing procedures of a voice control process executed by the voice control program601inFIG.6. The voice control process inFIG.13is achieved because the CPU302of the voice control apparatus100develops the voice control program601stored in the external storage unit305onto the RAM303and runs it.

As shown inFIG.13, the voice control program601obtain the voice that the user utters to the microphone308with the voice obtainment module604(step S1301). The voice control program601determines an end of the obtained voice with the utterance end determination module608, converts the voice from an utterance start to an utterance end into voice data, and stores the voice data concerned into the external storage unit305. Next, the voice control program601determines whether the voice data is detected (step S1302). In the step S1302, when the process for storing the voice data into the external storage unit305is completed, the voice control program601determines that the voice data is detected, for example. In the meantime, when the utterance end determination module608does not determine the end of the obtained voice and the process for storing the voice data into the external storage unit305is not completed, the voice control program601determines that the voice data is not detected.

As a result of the determination in the step S1302, when the voice data is not detected, the voice control process returns to the step S1301. As a result of the determination in the step S1302, when the voice data is detected, the voice control program601blinks the LED312with the display module606(step S1303) to indicate that the voice control apparatus100is in the response processing state. Next, the voice control program601transmits the voice data stored in the external storage unit305to the server102with the data transmission/reception module602(step S1304). After that, the voice control program601waits until receiving the combined voice data from the server102.

When the combined voice data is received from the server102(YES in the step S1305), the voice control program601reproduces the received combined voice data with the voice reproduction module605(step S1306). Next, the voice control program601determines whether an interactive-session-end notification is received from the server102(step S1307).

As a result of the determination in the step S1307, when the interactive-session-end notification is not received from the server102, the voice control process returns to the step S1301. As a result of the determination in the step S1307, when the interactive-session-end notification is received from the server102, the voice control program601turns off the LED312with the display module606(step S1308) to indicate that the voice control apparatus100is in the standby state. Next, the voice control program601closes an interactive session (step S1309) and the voice control process is finished. After finishing the voice control process, when the user utters to the microphone308, the process of the step S1301will be executed.

FIG.14is a flowchart showing procedures of a voice recognition control process executed by the voice recognition program701inFIG.7. The voice recognition control process inFIG.14is achieved because the CPU402aof the server102develops the voice recognition program701stored in the external storage unit405aonto the RAM403aand runs it.

As shown inFIG.14, the voice recognition program701receives voice data or text data with the data transmission/reception module702(step S1401). The voice recognition program701stores the received data and the information about the transmitting source of the data concerned into the external storage unit405awith the data management module703. Next, the voice recognition program701determines whether the data stored in the external storage unit405ais voice data (step S1402).

As a result of the determination in the step S1402, when the data stored in the external storage unit405ais voice data, the voice recognition program701applies the voice recognition process to the received voice data with the voice recognition module705(step S1403). Moreover, the voice recognition program701stores the result of the voice recognition process into the external storage unit405awith the data management module703. Next, the voice recognition program701performs morphological analysis to the stored result of the voice recognition process with the morphological analysis module706to generate text data. The voice recognition program701transmits this text data to the remote control program801(step S1404), and the voice recognition control process is finished.

As a result of the determination in the step S1402, when the data stored in the external storage unit405ais not voice data, the voice recognition program701determines whether the data stored in the external storage unit405ais response text data (step S1405).

As a result of the determination in the step S1405, when the data stored in the external storage unit405ais the response text data, the voice recognition program701applies the voice combination process to the data stored in the external storage unit405awith the voice combination module707to generate combined voice data (step S1406). Next, the voice recognition program701transmits the generated combined voice data to the voice control apparatus100(step S1407), and the voice recognition control process is finished.

As a result of the determination in the step S1405, when the data stored in the external storage unit405ais not response text data, the voice recognition program701determines that the data stored in the external storage unit405ais invalid data. The voice recognition program701generates the combined voice data that reproduces the error message “voice cannot be recognized” (step S1408). After that, the voice recognition control process proceeds to the step S1407.

FIG.15is a flowchart showing procedures of a remote control process executed by the remote control program801inFIG.8A. The remote control process inFIG.15is achieved because the CPU402bof the server102develops the remote control program801stored in the external storage unit405bonto the RAM403band runs it.

As shown inFIG.15, the remote control program801receives text data from the voice recognition program701with the data transmission/reception module802(step S1501). The remote control program801stores this text data into the external storage unit405bwith the data management module803. Specifically, the remote control program801compares the received text data with the voice recognition messages that the data management module803manages. The remote control program801determines whether the voice recognition messages that the data management module803manages include the voice recognition message that matches the received text data (step S1502).

As a result of the determination in the step S1502, when the voice recognition messages that the data management module803manages include the voice recognition message that matches the received text data, the remote control program801transmits the operation instruction corresponding to this voice recognition message to the image forming apparatus101with the data transmission/reception module802(step S1503). When receiving the operation instruction, the image forming apparatus101performs the process in the above-mentioned step S1203to control the display of the touch panel in accordance with the received operation instruction. Moreover, the image forming apparatus101performs the process in the above-mentioned step S1204to transmit the screen information that shows the type of the screen currently displayed on the touch panel200and the UI widget information that shows the UI widgets included in the screen concerned to the server102.

The remote control program801receives the screen information and UI widget information from the image forming apparatus101with the data transmission/reception module802(step S1504). Next, the remote control program801specifies one or more voice recognition messages corresponding to the specified UI widget information from among the voice recognition messages that the data management module803manages. The remote control program801transmits all the specified voice recognition messages to the image forming apparatus101with the data transmission/reception module802(step S1505). Next, the remote control process proceeds to a step S1508mentioned later.

As a result of the determination in the step S1502, when the voice recognition messages that the data management module803manages do not include the voice recognition message that matches the received text data, the remote control program801compares the received text data with the filter words that the data management module803manages. The remote control program801determines whether the filter words that the data management module803manages include the filter word that matches the received text data (step S1506).

As a result of the determination in the step S1506, when the filter word that matches the received text data is included in the filter words that the data management module803manages, the remote control program801transmits the filter display instruction to the image forming apparatus101(step S1507). As mentioned above, this filter display instruction includes the voice recognition message of which the filter word matches the received text data and the UI widget information corresponding to the voice recognition message concerned.

Next, when receiving the response to the voice operation information like the operation instruction or the filter display instruction from the image forming apparatus101(step S1508), the remote control program801determines whether the received response is the screen update notification (step S1509).

As a result of the determination in the step S1509, when the received response is the screen update notification, the remote control program801stores the screen information included in the received screen update notification into the external storage unit405bwith the data management module803(step S1510). Next, the remote control program801transmits the response text data to the voice recognition program701(step S1511), and the remote control process is finished.

As a result of the determination in the step S1509, when the received response is not the screen update notification, the remote control program801determines whether the received response is the voice operation failure response showing that the impracticable instruction is received (step S1512).

As a result of the determination in the step S1512, when the received response is the voice operation failure response, the remote control program801transmits a response text data including a message showing that the voice operation corresponding to the received voice operation instruction was failed to the voice recognition program701(step S1513). For example, the message “voice operation was impossible” is included. After that, the remote control process is finished.

When the received response is not the voice operation failure response as a result of the determination in the step S1512, or when the filter word that matches the received text data is not included in the filter words that the data management module803manages as a result of the determination in the step S1506, the remote control program801transmits an invalid data response to the voice recognition program701(step S1514). The response shows that the voice operation corresponding to the text data received in the step S1501is invalid. After that, the remote control process is finished.

According to the above-mentioned embodiment, the text data is output on the basis of the voice data that is obtained by coding the voice obtained by the voice control apparatus100, and the predetermined process linked to the UI widget is executed on the basis of the voice recognition message and text data. The voice recognition message includes a word related to the predetermined process linked to the UI widget at least. And the voice recognition message is displayed in association with the UI widget included in the screen that is currently displayed on the touch panel200of the image forming apparatus101. That is, the voice recognition message that has the semantic relation with the process executed by the function corresponding to the UI widget is displayed in association with the UI widget. Thereby, the user can easily master a relation between an execution process and an utterance instruction.

Moreover, in the above-mentioned embodiment, when the text data does not match the voice recognition message, the filter display of the UI widgets relevant to the text data is performed and the voice recognition messages are displayed in association with the UI widgets concerned. This facilitates finding of a UI widget corresponding to a desired function by narrowing displayed UI widgets to UI widgets relevant to the text data on the basis of the voice recognition messages displayed in association with the UI widgets, while keeping the ease of mastery of a relation between an execution process and an utterance instruction.

In the above-mentioned embodiment, since a UI widget corresponding to each process is an icon, a mark, a button, an arrow, or a tab displayed on the touch panel200, a user can easily master a relation between each process and an utterance instruction.

Moreover, in the above-mentioned embodiment, a voice recognition message is displayed in a balloon (see the balloon1101, for example) displayed in association with a UI widget. Thereby, the user can easily understand a relation between a UI widget and a voice recognition message.

In the above-mentioned embodiment, the balloon1102showing that the voice operation is impossible is displayed in association with the secure print button206corresponding to the function that does not allow the voice operation among the UI widgets included in the screen currently displayed on the touch panel200. Thereby, the user can easily understand that the voice operation of the secure print206is impossible.

In the above-mentioned embodiment, a predetermined process linked to a UI widget is a print process, a setting acceptance process about the print process, a scan process, or a setting acceptance process about the scan process. Thereby, the user can easily perform voice operations of acceptance of setting and an execution instruction about the print process and scan process.

Although the present invention has been described using the above-mentioned embodiment, the present invention is not limited to the above-mentioned embodiment. For example, the data management module803may not manage the filter words. In such a case, UI widgets that are subjected to the filter display may be specified on the basis of related words (a synonym, similar word, Web retrieval result) of a voice recognition message.

Moreover, the voice recognition program701and remote control program801may be stored in the external storage unit505of the image forming apparatus101. In such a case, the image forming apparatus101executes the voice recognition control process inFIG.14and the remote control process inFIG.15. In such a configuration, the image forming apparatus101obtains the voice data that is obtained by coding the voice of the user106from the microphone provided in the voice control apparatus100or the image forming apparatus101. In this way, a user can easily master a relation between an execution process and an utterance instruction in the configuration that the image forming apparatus101recognizes voice without using the server102.

In the above-mentioned embodiment, the home screen of the image forming apparatus101is not restricted to the home screen201, and another home screen is also controlled similarly.

FIG.16is a view showing an example of the other home screen201bdisplayed on the touch panel200inFIG.2. The home screen202bcorresponds to the tab 3. The home screen202bincludes the UI widgets relevant to the scan function of the image forming apparatus101. Specifically, an often-use-setting-A button1601, an often-use-setting-B button1602, a send-with-fixed-phrase button1603, a fixed destination scan button1604, and a sorting scan button1605are included.

The often-use-setting-A button1601is a UI widget for designating execution of a process that scans a document on the basis of a scan setting registered beforehand to generate image data and sends the image data concerned.

The often-use-setting-B button1602is a UI widget for designating execution of a process that scans a document on the basis of a scan setting that is different from the scan setting for the often-use-setting-A button1601to generate image data and sends the image data concerned. The send-with-fixed-phrase button1603is a UI widget for designating execution of a process that scans a document to generate image data, attaches the image data to text information registered beforehand of a body of an E-mail, and sends the E-mail. The text information of the email body registered beforehand is a fixed phrase “We attach the data. Thank you for your assistance.”, for example.

The fixed destination scan button1604is a UI widget for designating execution of a process that scans a document to generate image data and sends the image data concerned to a destination Z registered beforehand. The sorting scan button1605is a UI widget for designating execution of a process that scans a document to generate image data and saves the image data concerned as a file name of the scanned date and time etc.

When the user utters “start voice operation” to the microphone308of the voice control apparatus100in a state where the home screen201bis displayed on the touch panel200, voice recognition messages are displayed in association with the respective buttons of the home screen201bas shown inFIG.16.

Moreover, in the above-mentioned embodiment, the voice recognition messages and filter words may be set using a Web server function of the remote control program801.

FIG.17is a view showing an example of a setting screen1700for setting up voice recognition messages and filter words in the embodiment. As shown inFIG.17, the home screen201corresponding to the tab 1, the home screen201acorresponding to the tab 2, the home screen201bcorresponding to the tab 3, and text boxes corresponding to them are displayed. The setting screen1700is displayed by accessing the remote control program801of the server102by a Web browser. Names of the UI widgets (buttons) included in a corresponding home screen are set to a position column of a text box. The user can set a voice recognition message and a filter word in association with a name of each button. It should be noted that a symbol “-” is beforehand set in each of the voice recognition message field and the filter word field corresponding to the name of the UI widget like the secure print button206that does not allow the voice operation. A plurality of filter words can be set by inserting a delimiter “,” between words. Moreover, the user can change the screen information and a UI widget message in the setting screen1700.

FIG.18is a flowchart showing procedures of a setting control process executed by the remote control program801inFIG.8A. The setting control process inFIG.18is achieved because the CPU402bof the server102develops the remote control program801stored in the external storage unit405bonto the RAM403band runs it.

As shown inFIG.18, the remote control program801receives data with the data transmission/reception module802(step S1801). Next, the remote control program801determines whether the received data is text data transmitted from the voice recognition program701(step S1802).

As a result of the determination in the step S1802, when the received data is the text data transmitted from the voice recognition program701, the remote control program801executes the remote control process inFIG.15(step S1803) and the setting control process is finished.

As a result of the determination in the step S1802, when the received data is not the text data transmitted from the voice recognition program701, the remote control program801determines whether the received data is the access notification transmitted from the Web browser of the client terminal103etc. (step S1804).

As a result of the determination in the step S1804, when the received data is not the access notification transmitted from the Web browser of the client terminal103etc., the remote control program801transmits the invalid response showing that the process is impossible to the transmitting source of the received data (step S1805). After that, the setting control process is finished.

As a result of the determination in the step S1804, when the received data is the access notification transmitted from the Web browser of the client terminal103etc., the remote control program801obtains the information that the data management module803manages (step S1806). Specifically, the remote control program801obtains the screen information, UI widget information, voice recognition messages, and filter words that the data management module803manages. Next, the remote control program801generates setting screen display data for displaying the setting screen1700on the Web browser of the client terminal103on the basis of the information obtained in the step S1806(step S1807). Next, the remote control program801transmits the generated setting screen display data to the client terminal103(step S1808). The client terminal103displays the setting screen1700on the Web browser of the client terminal103on the basis of the received setting screen display data. The client terminal103transmits the setting data that the user set up in the setting screen1700to the server102. The setting data includes a voice recognition message and a filter word that are changed by the user so that the corresponding screen information and UI widget information can be distinguished. Moreover, the setting data includes screen information about a new screen set by the user and UI widget information corresponding to the screen information concerned.

Next, the remote control program801receives the setting data from the client terminal103(step S1809). Next, the remote control program801specifies changed data on the basis of the setting data and the information obtained in the step S1806and obtains the specified changed data (step S1810). When plural changed data are specified, the remote control program801obtains one changed data from among the plural changed data in the step S1810. Next, the remote control program801determines whether the obtained changed data is a voice recognition message (step S1811).

As a result of the determination in the step S1811, when the obtained change data is a voice recognition message, the remote control program801specifies the voice recognition message of which the screen information and UI widget information match that of the change data from among the voice recognition messages that the data management module803manages. The remote control program801changes the specified voice recognition message to the changed data (step S1812). Next, the setting control process proceeds to a step S1817mentioned later.

As a result of the determination in the step S1811, when the changed data is not a voice recognition message, the remote control program801determines whether the obtained changed data is a filter word (step S1813).

As a result of the determination in the step S1813, when the obtained change data is a filter word, the remote control program801specifies the filter word of which the screen information and UI widget information match that of the change data from among the filter words that the data management module803manages. The remote control program801changes the specified filter word to the changed data (step S1814). Next, the setting control process proceeds to the step S1817mentioned later.

As a result of the determination in the step S1813, when the changed data is not a filter word, the remote control program801determines whether the obtained changed data is the screen information or UI widget information (step S1815).

As a result of the determination in the step S1815, when the changed data is not the screen information nor the UI widget information, the setting control process is finished. As a result of the determination in the step S1815, when the changed data is the screen information or the UI widget information, the remote control program801specifies the information corresponding to the changed data from among the plurality of pieces of screen information and UI widget information that the data management module803manages. The remote control program801changes the specified information to the changed data (step S1816). Next, the remote control program801determines whether the processes for all the changed data are completed (step S1817).

As a result of the determination in the step S1817, when the process for any changed data is not completed, the setting control process returns to the step S1801. As a result of the determination in the step S1817, when the processes for all the changed data are not completed, the remote control program801transmits the changed information to the image forming apparatus101(step S1818). The image forming apparatus101changes the display of the home screen on the basis of the received information. After that, the setting control process is finished.

The user can easily customize the voice recognition message and filter word by performing the above-mentioned setting control process inFIG.18.

Although the configuration where the data management module803of the remote control program801manages the voice recognition messages and filter words is described in the above-mentioned embodiment, the present invention is not restricted to this configuration. For example, the data management module903of the device control program901may manage the voice recognition messages and filter words. When the data management module903manages the voice recognition messages and filter words, the user can change the settings of the voice recognition messages and filter words on the touch panel200of the image forming apparatus101.

Moreover, when the data management module903of the device control program901manages the voice recognition messages and filter words, the device control program901may execute the setting control process inFIG.18using the Web server function of the device control program901.

A CPU is an acronym of a Central Processing Unit. A DNN is an acronym of a Deep Neural Network. A GMM is an acronym of a Gaussian Mixture Model. An HDD is an acronym of a Hard Disk Drive. An HMM is an acronym of a Hidden Markov Model. An ID is an abbreviation of an Identification. An IEEE is an acronym of an Institute of Electrical and Electronics Engineers. An IP is an acronym of an Internet Protocol. A LAN is an acronym of a Local Area Network. An LCD is an acronym of a Liquid Crystal Display. An LED is an acronym of a Light Emitting Diode. An MEMS is an acronym of Micro Electro Mechanical Systems. An MP3 is an abbreviation of MPEG Audio Layer-3. A PC is an acronym of a Personal Computer. A RAM is an acronym of a Random Access Memory. An RNN is an acronym of Recurrent Neural Networks. A ROM is an acronym of a Read Only Memory. An SD card is abbreviation of a Secure Digital Memory Card. An SSD is an acronym of a Solid State Drive. A TCP is an acronym of a Transmission Control Protocol. A UI is an acronym of a User Interface. A URL is an acronym of a Uniform Resource Locator.

OTHER EMBODIMENTS

Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No. 2020-209345, filed Dec. 17, 2020, which is hereby incorporated by reference herein in its entirety.