Image forming system allowing voice operation, control method therefor, and storage medium storing control program therefor

An image forming system that is capable of reducing time and effort of a user for setting a display language. The image forming system including the following members. An image forming device forms an image on a sheet. A display device displays information. A microphone obtains voice. An obtainment unit obtains a plurality of pieces of word information based on audio information on the phrase obtained through the microphone. A specification unit specifies a language using the plurality of word information. An update unit updates a display language of the display unit based on the language specified by the specification unit.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image forming system that allows a voice operation, a control method therefor, and a storage medium that stores a control program therefor.

Description of the Related Art

An image forming apparatus like a printer that cooperates with a smart speaker is known (for example, see Japanese Laid-Open Patent Publication (Kokai) No. 2019-18394 (JP 2019-18394A)). A user can perform various settings of the image forming apparatus by inputting voice into the smart speaker. The image forming apparatus may be shared by a plurality of users who use different languages in an office, for example. Accordingly, a comfortable operating environment differs for every user. A user changes a language used in the image forming apparatus, such as a display language of a display device of the image forming apparatus, by inputting voice to the smart speaker.

However, since a user is required to change the setting of a display language of the display device whenever the user starts to use the conventional image forming apparatus, it takes time and effort.

SUMMARY OF THE INVENTION

The present invention provides an image forming system, a control method therefore, and a storage medium storing a control program therefor, which are capable of reducing time and effort of a user for setting a display language.

Accordingly, a first aspect of the present invention provides An image forming system including an image forming device configured to form an image on a sheet, a display device configured to display information, a microphone configured to obtain voice, an obtainment unit configured to obtain a plurality of pieces of word information based on audio information on the phrase obtained through the microphone, a specification unit configured to specify a language using the plurality of word information, and an update unit configured to update a display language of the display unit based on the language specified by the specification unit.

Accordingly, a second aspect of the present invention provides a control method for the image forming system having the image forming device, the display device, and the microphone, the control method including obtaining a plurality of pieces of word information based on audio information on a phrase obtained through the microphone, specifying a language using the plurality of pieces of word information, and updating a display language of the display device based on the specified language.

Accordingly and a third aspect of the present invention provides a non-transitory computer-readable storage medium storing a control program causing a computer to execute the control method of the second aspect.

According to the present invention, the time and effort of a user for setting a display language is reducible.

DESCRIPTION OF THE EMBODIMENTS

Hereafter, embodiments according to the present invention will be described in detail by referring to the drawings. It should be noted that the following embodiment does not restricts the invention according to the claims and all combinations of characteristic features described in the embodiment are not always indispensable to the solution of the invention.

FIG. 1is a configuration diagram showing an image forming system100according to the embodiment of the present invention. As shown inFIG. 1, the image forming system100is provided with an MFP (Multi-Function Peripheral)101as an image forming apparatus, a smart speaker102as a voice obtainment device, and a cloud server103. The MFP101and smart speaker102are connected to a network104. And the cloud server103is connected to the network104through a gateway105. Thereby, the MFP101, smart speaker102, and cloud server103are communicable through the network104.

The image forming system100is able to control the MFP101to execute a process corresponding to a user's voice operation that the smart speaker102obtains. For example, when a user gives a copy job execution instruction, such as “copy this”, the smart speaker102transmits audio data (audio information) corresponding to the copy job execution instruction to the cloud server103through the network104. When receiving the audio data, the cloud server103generates device operation data corresponding to the audio data and transmits the device operation data to the MFP101through the network104. The MFP101executes a copy job as a process corresponding to the received device operation data and transmits a response, which indicates that the copy job has been executed, to the cloud server103through the network104. When receiving the response, the cloud server103generates response message data and transmits the response message data to the smart speaker102through the network104. The smart speaker102outputs an audio message of “Now copying” that corresponds to the received response message data.

The MFP101is a multifunction apparatus equipped with a plurality of functions, such as a print function and a scan function. The MFP101is provided with MFP's own apparatus data106and MFP's other apparatus data107. The MFP's own apparatus data106includes an IP address and a MAC address of the MFP101that are used in data communications through the network104. The MFP's other apparatus data107includes account information used when the MFP101uses a service of the cloud server103and URL information about a response notification that notifies the cloud server103of an execution result of the process corresponding to the device operation data that is received from the cloud server103, for example.

The smart speaker102is a loudspeaker equipped with an audio assistant function and is provided with smart speaker's own apparatus data108and smart speaker's other apparatus data109. The smart speaker's own apparatus data108includes an IP address and a MAC address of the smart speaker102that are used in data communications through the network104. The smart speaker's other apparatus data109includes account information used when the smart speaker102uses a service of the cloud server103and a service URL of the cloud server103corresponding to a wake word mentioned later.

The cloud server103is provided with cloud server's own apparatus data110and cloud server's other apparatus data111. The cloud server's own apparatus data110includes service URL information used when the MFP101or the smart speaker102uses a service of the cloud server through the network104and the above-mentioned URL information about a response notification. The cloud server's other apparatus data111includes account information issued to the MFP101and the smart speaker102, and IP addresses and MAC addresses of the MFP101and smart speaker102. The cloud server103communicates with the MFP101and smart speaker102through the network104by using the IP addresses and MAC addresses included in the cloud server's other apparatus data111.

Various kinds of data, such as audio data that the smart speaker102generates and device operation data that the cloud server103generates, are transmitted and received through the network104. The gateway105is a wireless LAN router based on IEEE802.11 standards, such as IEEE802.11a and IEEE802.11b, for example. It should be noted that the gateway105may be a configuration based on a wireless communication standard other than the IEEE802.11 standards. Moreover, the gateway105may be a wired LAN router based on Ethernet standards, such as 10BASE-T, 100BASE-T, and 1200BASE-T.

FIG. 2is a block diagram schematically showing a hardware configuration of the MFP101inFIG. 1. As shown inFIG. 2, the MFP101is provided with a controller200, an operation panel209, a print engine (an image forming device)211, and a scanner213. The controller200is connected with the operation panel209, print engine211, and scanner213. Moreover, the controller200is provided with a CPU (Central Processing Unit)202, a RAM203, a ROM204, a storage unit205, a network I/F206, a display controller207, an operation I/F208, a print controller210, and a scan controller212. The CPU202, RAM203, ROM204, storage unit205, network I/F206, display controller207, operation I/F208, print controller210, and scan controller212are mutually connected through a system bus201.

The CPU202controls operations of the entire MFP101. The CPU202reads a control program stored in the ROM204or the storage unit205and performs various control processes, such as a reading control process and a printing control process. The RAM203is a main memory of the CPU202. The RAM203is used as a work area of the CPU202and as a temporary storage area to which the control program stored in the ROM204or the storage device205will be developed. The ROM204stores the control program that the CPU202runs. The storage unit205stores print data, image data, programs, setting information, etc. Although the MFP101of the embodiment is configured so that the single CPU202will execute processes mentioned later using a single memory (the RAM203), the configuration of the MFP101is not restricted to this configuration. For example, the MFP101may be configured so that a plurality of CPUs, RAMs, ROMs, and storage units will cooperatively execute the processes mentioned later. Moreover, the MFP101may execute some processes using a hardware circuit, such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

The network I/F206is used when the MFP101communicates with another apparatus through the network104. For example, the MFP101analyzes print data received through the network I/F206by a PDL analysis module (not shown). The PDL analysis module is a software module for analyzing print data and generates image data, which is printed by the print engine211, on the basis of the print data expressed by various kinds of page description languages. A program for booting the PDL analysis module is stored in the storage unit205or the ROM204.

The display controller207and operation I/F208are connected with the operation panel209. The operation I/F208performs display control of a screen of the operation panel209. When a user operates the operation panel209, the MFP101obtains an event corresponding to the user's operation through the display controller207.

The print controller210is connected with the print engine211. The print controller210transfers the image data that is generated by the above-mentioned PDL analysis module to the print engine211. The print engine211forms the received image data on a sheet. An electrophotographic system, an ink jet system, or the like is used as a printing system of the print engine211. When the electrophotographic system is used, an image is formed on a sheet by developing an electrostatic latent image formed on a photosensitive member, transferring a developed toner image to a sheet, and fixing the transferred toner image. When the ink jet system is used, an image is formed on a sheet by discharging ink.

The scan controller212is connected with the scanner213. The scanner213reads an image on a sheet and generates image data. The image data generated by the scanner213is stored in the storage unit205. Moreover, an image is formed on a sheet using the image data that the scanner213generates. The scanner213has a document feeder (not shown) and can read documents stacked on the document feeder by conveying them one by one.

FIG. 3is a block diagram schematically showing a hardware configuration of the smart speaker inFIG. 1. As shown inFIG. 3, the smart speaker102is provided with a controller300, a microphone308, a loudspeaker310, and an LED312. The controller300is connected with the microphone308, loudspeaker310, and LED312. Moreover, the controller300is provided with a CPU302, a RAM303, a ROM304, a storage unit305, a network I/F306, a microphone I/F307, an audio controller309, and a display controller311. The CPU302, RAM303, ROM304, storage unit305, network I/F306, microphone I/F307, audio controller309, and display controller311are mutually connected through a system bus301.

The CPU302is a central processing unit that controls operations of the entire controller300. The RAM303is a volatile memory. The ROM304is a nonvolatile memory and stores a boot program of the CPU302. The storage unit305is a storage device with a larger memory capacity than the RAM303and may be an SD card. It should be noted that the storage unit305may be a flash ROM, not the SD card, or may be another storage device having a function equivalent to that of the SD card. For example, the storage unit305stores a control program of the smart speaker102that the controller300executes.

When booting the smart speaker102in response to a power ON operation by a user, the CPU302runs a boot program stored in the ROM304. The boot program reads a control program stored in the storage unit305and develops the control program concerned onto the RAM303. The CPU302runs the control program developed onto the RAM303and performs various control processes. Moreover, the CPU302stores data used when running the control program into the RAM303or the storage unit305. The CPU302communicates with another apparatus on the network104through the network I/F306.

The network I/F306includes a circuit and an antenna that enable communication according to the wireless communication system based on IEEE802.11 standard. It should be noted that the network I/F306may employ a cable communication system based on the Ethernet standard, not the wireless communication system. The microphone I/F307is connected to the microphone308. The microphone I/F converts user's voice received by the microphone308into coded audio data and stores the converted audio data into the RAM303in accordance with an instruction from the CPU302.

The microphone308is a compact MEMS microphone mounted in a smart phone etc., for example. It should be noted that the microphone308is not limited to the MEMS microphone and may be another device that can obtain user's voice. In the embodiment, it is preferable to arrange three or more microphones308at predetermined positions in order to specify an arrival direction of user's voice.

The audio controller309is connected to the loudspeaker310. The audio controller309converts the audio data into an analog voice signal in accordance with an instruction from the CPU302and outputs sound through the loudspeaker310.

The loudspeaker310reproduces an audio response indicating that the smart speaker102is responding and also reproduces sound composited by the cloud server103. The loudspeaker310is a general-purpose device for reproducing sound.

The display controller311is connected to the LED312. The display controller311controls light emission of the LED312in accordance with an instruction from the CPU302. In the embodiment, the display controller311controls the light emission of the LED312to indicate that the smart speaker102is obtaining user's voice. The LED312is, for example, a blue LED visible to a user. The LED312is a general-purpose device. In the embodiment, the smart speaker102may be provided with a display device capable of displaying text and picture indicating that the smart speaker102is obtaining user's voice instead of the light emission of the LED312.

FIG. 4is a block diagram schematically showing a hardware configuration of a controller400of the cloud server103inFIG. 1. As shown inFIG. 4, the controller400is provided with a CPU402, RAM403, ROM404, storage unit405, and network I/F406. The CPU402, RAM403, ROM404, storage unit405, and network I/F406are mutually connected through a system bus401.

The CPU402is a central processing unit that controls operations of the entire controller300. The RAM403is a volatile memory. The ROM404is a nonvolatile memory and stores a boot program of the CPU402, etc. The storage unit405is a storage device with a larger memory capacity than the RAM403and may be a hard disk drive (HDD). It should be noted that the storage unit405may be a solid state drive (SSD) or may be another storage device having a function equivalent to that of the HDD. The storage unit405stores a control program of the cloud server103that the controller400executes, for example.

The CPU402runs a boot program stored in the ROM404when booting the cloud server103. The boot program reads the control program stored in the storage unit405and develops the control program concerned onto the RAM403. The CPU402runs the control program developed onto the RAM403and performs various control processes. Moreover, the CPU402stores data used when running the control program into the RAM403or the storage unit405. The CPU402communicates with another apparatus on the network104through the network I/F406.

FIG. 5is a block diagram schematically showing a configuration of a device control module500as a software module of the MFP101inFIG. 1. As shown inFIG. 5, the device control module500includes a data transmission/reception module501, a data analysis module502, a job control module503, a data management module504, a display module505, an operation target determination module506, a scan module507, and a print module508. A process executed by these modules is achieved because the CPU202runs the control program developed from the ROM204to the RAM203.

The data transmission/reception module501controls transmission and reception of data between the MFP101and another apparatus on the network104through the network I/F206in accordance with TCP/IP. For example, the data transmission/reception module501controls reception of device operation data that the cloud server103generates. Moreover, the data transmission/reception module501controls transmission of various notifications from the MFP101to the cloud server103. The various notifications include a notification indicating a job execution result and a notification indicating a job execution status, for example.

The data analysis module502converts the device operation data received by the data transmission/reception module501into commands that the modules of the device control module500can interpret and transmits corresponding commands to the job control module503, data management module504, and display module505.

The job control module503gives instructions to the print controller210and scan controller212to respectively control the print engine211and scanner213. The data management module504stores the data about the process by the device control module500to predetermined areas of the RAM203and storage unit205and manages the data. The data about the process by the device control module500includes, for example, job data that is a combination of a setting item and set value of a job executed by the job control module503, and a language setting data that shows a language of texts displayed on the operation panel209. Moreover, the data management module504stores authentication information that is needed for communication with the gateway105, the device information that is needed for communication with the cloud server103, etc. into the RAM203or the storage unit205and manages the information. Furthermore, the data management module504stores screen control information that the display module505uses for the display control of the screen, and operation target determination information that the operation target determination module506uses to determine an operation target. The screen control information and operation target determination information are managed for every screen that the display module505displays.

The display module505gives an instruction about the display control of the operation panel209to the display controller207. For example, when receiving the instruction from the display module505, the display controller207displays user operatable UI members (buttons, a pulldown list, a check box, etc.) on the operation panel209. The screen is updated on the basis of the screen control information. For example, the display module505obtains a language dictionary corresponding to the language setting data that the data management module504manages from the storage unit205and displays the text data generated on the basis of the language dictionary on the operation panel209.

The operation target determination module506obtains a coordinate showing a position that a user touches the operation panel209through the operation I/F208and determines an UI member that is displayed on the operation panel209at the position that the user touched as an operation target. The operation target determination module506reads the screen control information corresponding to the UI member determined as the operation target and determines the contents of the process on the basis of the screen control information concerned. The operation target determination module506instructs the modules of the device control module500to execute the determined process. For example, the operation target determination module506instructs the display module505to update the display contents of the screen and instructs the job control module503to start a job using the job parameters set by the user operation.

The scan module507controls the scanner213to execute a scan through the scan controller212on the basis of the scan setting received from the job control module503and controls the data management module504to store the read image data. The print module508controls the print engine211to print via the print controller210on the basis of the print setting received from the job control module503.

FIG. 6is a block diagram schematically showing a configuration of an audio control module600as a software module of the smart speaker102inFIG. 1. As shown inFIG. 6, the audio control module600includes a data transmission/reception module601, a data management module602, a control module603, a voice obtainment module604, an audio reproduction module605, a display module606, a voice-operation start detection module607, and an utterance end determination module608. A process executed by these modules is achieved because the CPU302runs the control program developed from the storage unit305to the RAM303.

The data transmission/reception module601controls transmission and reception of data between the smart speaker102and another apparatus on the network104through the network I/F306in accordance with TCP/IP. For example, the data transmission/reception module601controls the transmission of audio data of user's voice, which is obtained by the voice obtainment module604, to the cloud server103. Moreover, the data transmission/reception module601controls the reception of composite audio data (mentioned later) from the cloud server103.

The data management module602stores the data related to the process by the audio control module600to a predetermined area of the storage unit305. The data about the process by the audio control module600includes, for example, sound volume setting data of a sound reproduced by the audio reproduction module605, authentication information that is needed for communication with the gateway105, and device information that is needed for communication with the MFP101and cloud server103.

The voice obtainment module604generates audio data by converting user's analog voice picked up by the microphone308into a digital signal in a predetermined format like MP3 and by coding the digital signal and stores the audio data concerned into the RAM303temporarily. The control module603manages start and end timings of the process by the voice obtainment module604. It should be noted that the format of the audio data may be a general-purpose streaming format. The coded audio data may be transmitted to the data transmission/reception module601subsequently.

The audio reproduction module605controls the audio controller309to reproduce composite audio data (audio message) received by the data transmission/reception module601using the loudspeaker310. The control module603manages an execution timing of the audio reproduction process by the audio reproduction module605.

The display module606controls the light emission of the LED312through the display controller311. For example, when the voice-operation-start detection module607detects a voice operation, the display module606lights s the LED312through the display controller311. The control module603manages an execution timing of the process by the display module606.

When detecting a wake word that a user utters or a press operation of an operation start key (not shown) of the smart speaker102, the voice-operation-start detection module607transmits an operation start notification showing detection of the wake word or the press operation to the control module603. The wake word is a voice word for starting the audio assistant function of the smart speaker102and is registered beforehand. The voice-operation-start detection module607detects the wake word from user's analog voice picked up by the microphone308. The user can operate the MFP101by uttering a phrase corresponding to an instruction after uttering the wake word.

The utterance end determination module608determines an end timing of the process of the voice obtainment module604. For example, the utterance end determination module608determines that user's utterance finishes when a pause of user's voice reaches a predetermined period like three seconds. And then, the utterance end determination module608transmits the utterance end notification showing the determination result to the control module603. It should be noted that the end of user's utterance may be determined on the basis of utterance of a predetermined word registered beforehand in place of the no-utterance period (referred to as a “pause period”). For example, when the user utters predetermined words registered beforehand, such as “yes”, “no”, “OK”, “cancel”, “finish”, “start”, and “begin”, the utterance end determination module608may determine that user's utterance finishes without waiting for the predetermined period. Moreover, not the smart speaker102but the cloud server103may determine the end of utterance, and the cloud server103may determine that the user's utterance finishes on the basis of the meaning and context of contents of user's utterance.

The control module603controls the other modules in the audio control module600so as to operate in conjunction mutually. Specifically, the control module603controls starts and ends of the processes of the voice obtainment module604, audio reproduction module605, and display module606. Moreover, the control module603controls the data transmission/reception module601to transmit the audio data to the cloud server103after the voice obtainment module604obtains the audio data. Moreover, the control module603controls the audio reproduction module605to reproduce composite audio data after receiving the composite audio data from the cloud server103.

The start and end timings of the processes by the voice obtainment module604, audio reproduction module605, and display module606will be described.

When the operation start notification is received from the voice-operation-start detection module607, the control module603starts the process by the voice obtainment module604. Moreover, when receiving the utterance end notification from the utterance end determination module608, the control module603finishes the process by the voice obtainment module604. For example, when a user utters a wake word, the voice-operation-start detection module607detects the wake word and transmits the operation start notification to the control module603. When receiving the operation start notification, the control module603controls the voice obtainment module604to start the process. The voice obtainment module604obtains user's voice (for example, “I wants to copy”) following the wake word, converts the voice into audio data, and stores the audio data temporarily. The utterance end determination module608transmits the utterance end notification to the control module603, when the pause period of the predetermined period continues after the voice of “I wants to copy”. When receiving the utterance end notification, the control module603controls the voice obtainment module604to finish the process. Hereinafter, the state between the start and end of the process by the voice obtainment module604will be referred to as an “utterance processing state”. The display module606lights the LED312as a notification to indicate being in the utterance processing state.

After determining that the user's utterance finished, the control module603instructs the data transmission/reception module601to transmit the audio data that is temporarily stored in the voice obtainment module604to the cloud server103and waits for a response from the cloud server103. The response from the cloud server103includes, for example, a header section indicating the response and a response message that consists of the composite audio data. The control module603controls the audio reproduction module605to reproduce the composite audio data when the data transmission/reception module601receives the above-mentioned response. The composite audio data is an audio message of “copy screen will be displayed”, for example. It should be noted that the state between the determination that the user's utterance finished and the end of reproduction of the composite audio data will be referred to as a “response processing state”. The display module606blinks the LED312as a notification to indicate being in the response processing state.

After finishing reproduction of the composite audio data, the user is able to give an instruction by emitting a phrase corresponding to the instruction without uttering a wake word while the interactive session with the cloud server103is continuing. The end of the interactive session is determined when the cloud server103transmits an interactive session end notification to the smart speaker102. The state between the end of one interactive session and the start of another interactive session will be referred to as a “standby state”. That is, the smart speaker102is in the standby state until the control module603receives an operation start notification from the voice-operation-start detection module607. During the standby state, the display module606turns off the LED312as a notification to indicate being in the standby state.

FIG. 7A,FIG. 7B, andFIG. 7Care views for describing an audio data conversion control module as a software module700of the cloud server103inFIG. 1.FIG. 7Ais a block diagram schematically showing a configuration of the audio data conversion control module700.FIG. 7Bshows examples of Japanese group ID lists that are used by a group ID determination module707mentioned later for determining a group ID.FIG. 7Cshows examples of English group ID lists that are used by the group ID determination module707mentioned later for determining a group ID. In the group ID lists, words having the same meaning or intention in relation to user's operations to the MFP101are grouped under the same ID. It should be noted that the words listed here are results of the voice recognition of the words that the user utters to the smart speaker102. Moreover, in the group ID lists, a language determination exception flag that shows whether the language determination mentioned later is excepted is set to each registered word. In the group ID lists, YES is set to the language determination exception flag of a word, such as a “katakana” word like “kopi”, that is impossible to specify whether the word is in English or Japanese. The word to which YES is set to the language determination exception flag is not used for the language determination mentioned later. In the meantime, in the group ID lists, NO is set to the language determination exception flag of a word other than a katakana word. The word to which NO is set to the language determination exception flag is used for the language determination mentioned later.

InFIG. 7A, the audio data conversion control module700includes a data transmission/reception module701, a data management module702, a device-operation-data generation module703, and an audio data conversion module710. The audio data conversion module710includes a voice recognition module705, a morphological analysis module706, a group ID determination module707, and an audio composition module708. A process executed by the above-mentioned modules is achieved because the CPU402runs the control program developed from the storage unit405to the RAM403.

The data transmission/reception module701controls transmission and reception of data between the cloud server103and another apparatus on the network104through the network I/F406in accordance with TCP/IP. For example, the data transmission/reception module701receives user's audio data from the smart speaker102. Moreover, the data transmission/reception module701transmits the group ID that the group ID determination module707determined and the determination result of the text data by the voice recognition process executed by the voice recognition module705to the MFP101.

The data management module702stores the data related to the process of the audio data conversion control module700to a predetermined area of the storage unit405. The data related to the process of the audio data conversion control module700include, for example, an acoustic model and language model for converting audio data received by the data transmission/reception module701into text data, a dictionary that is used when the morphological analysis module706morphologically analyzes a text, the group ID lists that are used when the group ID determination module707determines a group ID, an audio database that is used when the audio composition module708performs an audio composition process, and device information needed to communicate with the smart speaker102or the MFP101.

The voice recognition module705performs the voice recognition process to convert the user's audio data received by the data transmission/reception module701into a text. The voice recognition process converts the user's audio data into phonemes using the acoustic model and also converts the phonemes into actual text data using the language model. It should be noted that the user's audio data may include words of several different languages. In the embodiment, the voice recognition process may employ a first voice recognition method that determines a language of input audio data and converts the audio data into text data in the determined language. Moreover, the voice recognition process may employ a second voice recognition method that converts input audio data to phonemes using acoustic models of a plurality of languages and converts the audio data into text data in each of the languages using the corresponding language model. Since the second voice recognition method converts audio data into text data in a plurality of language forms, the voice recognition module705generates voice recognition data that consists of a text and a language setting as an execution result of the voice recognition process.

In the embodiment, the languages of input voice are Japanese and English. Voice recognition data in Japanese is data that consists of the language setting “Japanese” and a text consisting of one or more kana. Voice recognition data in English is data that consists of the language setting “English” and a text consisting of one or more alphabet. It should be noted that the voice recognition process that converts audio data into voice recognition data is not restricted to the method mentioned above in the embodiment, and another method may be used.

The morphological analysis module706morphologically analyzes the voice recognition data converted by the voice recognition module705on the basis of the language setting. The morphological analysis module706deduces a morpheme string from a dictionary having information about grammar and parts of speech of the language and determines the part of speech of each morpheme (word information) that constitutes the morpheme string concerned. The morphological analysis module706can be achieved by using a well-known morphological analysis software, such as JUMAN, Web-Chamame, or MeCab.

An operation example of the morphological analysis module706will be described. For example, the morphological analysis module706analyzes voice recognition data {“yonbukopishite (four copies)”, the language setting “Japanese”} that is converted by the voice recognition module705as a morpheme string of “yon”, “bu”, “kopi”, “wo”, and “shite” Moreover, the morphological analysis module706analyzes voice recognition data {“Four Copies”, the language setting “English”} as a morpheme string of “Four” and “Copies”.

The group ID determination module707specifies group IDs by matching the result of the morphological analysis by the morphological analysis module706with the group ID lists corresponding to the language setting of the voice recognition data among the Japanese group ID lists inFIG. 7Band the English group ID lists inFIG. 7C. And then the group ID determination module707generates a group ID determination result indicating the specified group IDs. For example, the group ID determination module707matches the morpheme string of “yon”, “bu”, “kopi”, “wo”, and “shite” with the Japanese group ID lists inFIG. 7B, specifies “NUM00004”, “CNF00001”, and “FNC00001” that are the group IDs of “yon”, “bu”, and “kopi”, and generates {ID:NUM00004, ID:CNF00001, ID:FNC00001} as the group ID determination result. Moreover, the group ID determination module707matches the morpheme string of “Four” and “Copies” with the English group ID lists inFIG. 7C, specifies “NUM00004”, “CNF00001”, and “FNC00001” that are the group IDs of “Four” and “Copies”, and generates {ID:NUM00004, ID:CNF00001, ID:FNC00001} as the group ID determination result.

When the group ID determination result includes a plurality of group IDs, the group IDs are set up in the order of the results of the voice recognition and the morphological analysis. For example, when the results of the voice recognition and the morphological analysis are “yon”, “bu”, “kopi”, “wo”, and “shite”, the group IDs are set up in the order of {ID:NUM00004, ID:CNF00001, ID:FNC00001} corresponding to the morphemes “yon”, “bu”, and “kopi” as the group ID determination result. Moreover, when there are different group IDs corresponding to the same morpheme, the group ID determination result may include all the different group IDs. For example, “CNF00001” and “FNC00001” are associated with the same morpheme “copies” in the English group ID lists inFIG. 7C. When the results of the voice recognition and the morphological analysis are “four” and “copies”, the group ID determination result is generated as {ID:NUM00004, ID:CNF00001, ID:FNC00001}.

The audio composition module708performs an audio composition process on the basis of the notification received from the MFP101. In the audio composition process, a previously registered text that corresponds to the received notification is converted into audio data of a predetermined format, such as MP3. In the audio composition process, the audio data is generated on the basis of an audio database stored in the data management module702, for example. The audio database is, for example, a database that collects sounds of regular contents like words. Although the audio composition process is performed using the audio database in the embodiment, the method of the audio composition process is not restricted to this method. Another method may be used.

The device-operation-data generation module703determines the operation of the MFP101on the basis of the group ID determination result generated by the group ID determination module707and the language setting of the voice recognition data generated by the voice recognition module705. The device-operation-data generation module703generates a file of a predetermined data format corresponding to the determined operation.

For example, when the language setting of the voice recognition data is “Japanese” and the group ID determination result is {ID:NUM00004, ID:CNF00001, ID:FNC00001}, the device-operation-data generation module703determines that Japanese is set to the language setting of the MFP101on the basis of the “Japanese” and generates a character string {“language”: “Japanese”}. The device-operation-data generation module703determines to instruct the MFP101to perform a copy job on the basis of “FNC00001” and generates character strings {“operation”: “jobStart”} and {“jobName”: “copy”} for performing the copy job. The device-operation-data generation module703generates a character string {“copies”: “4”} for designating “4” as the number of copies of the copy job on the basis of “NUM00004” and “CNF00001”. The device-operation-data generation module703generates the data in the JSON format shown inFIG. 8by combining these character strings.

Moreover, when the language setting of the voice recognition data is “English” and the group ID determination result is {ID:FNC00001, ID:NUM00004, ID:CNF00002, ID:FNC00003}, the device-operation-data generation module703determines that English is set to the language setting of the MFP101on the basis of the “English” and generates a character string {“language”: “English”}. The device-operation-data generation module703determines to execute the job setting of the MFP101on the basis of “FNC00001” and “FNC00003” and generates a character string {“operation”: “jobSetting”} for executing the job setting. The device-operation-data generation module703generates a character string {“density”: “4”} on the basis of “NUM00004” and “CNF00001”. The device-operation-data generation module703generates the data in the JSON format shown inFIG. 9by combining these character strings.

FIG. 10is a sequence chart showing procedures of a process executed when the image forming system100ofFIG. 1receives a job execution instruction by voice input. It should be noted that the smart speaker102, MFP101, and cloud server103shall be communicable mutually inFIG. 10. Moreover, a home screen2001inFIG. 20on which functions, such as copy, scan, and print, can be called shall be displayed on the operation panel209of the MFP101.

InFIG. 10, a user gives an instruction to the smart speaker102to start a voice operation to in a step S1001first. The instruction to start the voice operation is given when a user utters the wake word or when the user presses an operation start key (not shown) of the smart speaker102. The instruction to start the voice operation is detected by the voice-operation-start detection module607.

When the instruction to start the voice operation is detected, the display module606of the audio control module600lights the LED312as a notification to indicate being in the utterance processing state in a step S1002in the smart speaker102. Moreover, the process by the voice obtainment module604is started in the smart speaker102.

In a step S1003, the user performs the function call instruction to the smart speaker102. For example, the user utters a phrase, such as “yonbukopishite” or “four copies”, that is a job execution instruction as a function call instruction following the wake word detected in the step S1001. Audio data is generated on the basis of user's voice that is obtained by the voice obtainment module604. When the pause period of the predetermined period continues, the utterance end determination module608determines that the utterance ends.

In a step S1004, the display module606of the audio control module600blinks the LED312as a notification to indicate being in the response processing state depending on the utterance end determination. Moreover, the process by the voice obtainment module604is completed. In a step S1005, the data transmission/reception module601transmits the generated audio data to the cloud server103.

In a step S1006, the audio data conversion control module700in the cloud server103executes the voice-operation-service execution process ofFIG. 11mentioned later. The details of the voice-operation-service execution process will be mentioned later. In the voice-operation-service execution process, the language-set job information that is device operation data for executing a job is transmitted to the MFP101, and the audio message mentioned later is transmitted to the smart speaker102, for example.

In a step S1007, the device control module500in the MFP101executes a language setting switching process ofFIG. 19mentioned later on the basis of the language-set job information received from the cloud server103.

In a step S1008, the data transmission/reception module601in the smart speaker102receives an audio message from the cloud server103. In the next step S1009, the audio reproduction module605reproduces the composite audio data into which the audio message received in the step S1008is converted. For example, the audio reproduction module605reproduces the composite audio data “copy will be started” through the loudspeaker310.

In a step S1010, the data transmission/reception module601receives an audio message that is different from the audio message received in the step S1008from the cloud server103. Moreover, the data transmission/reception module601receives the interactive session end notification that finishes then interactive session with the user from the cloud server103.

In a step S1011, the audio reproduction module605reproduces the composite audio data into which the audio message received in the step S1010is converted. For example, the audio reproduction module605reproduces the composite audio data “copy has been finished” through the loudspeaker310.

In a step S1012, the display module606turns off the LED312as a notification showing that the smart speaker102is in the standby state in response to the reception of the interactive session end notification by the data transmission/reception module601in the step S1010.

In a step S1013, the audio control module600finishes the interactive session and shifts the smart speaker102to the standby state in response to the reception of the interactive session end notification by the data transmission/reception module601in the step S1010.

In the sequence ofFIG. 10, even if the LED312is blinking as the notification to indicate being in the response processing state, the user can input a wake word into the smart speaker102. The interactive session may be compulsorily finished when the user utters “cancel” or “stop” after a wake word.

FIG. 11is a flowchart showing procedures of the voice-operation-service execution process executed by the cloud server103inFIG. 1. The voice-operation-service execution process is achieved because the CPU402runs the control program developed from the storage unit405to the RAM403. The voice-operation-service execution process ofFIG. 11is executed when the data transmission/reception module701receives, in the step S1005, the audio data of the function call instruction transmitted from the smart speaker102.

As shown inFIG. 11, the CPU402executes the voice recognition process that converts audio data into text data by the voice recognition module705(a step S1101). In the voice recognition process, the voice recognition module705may employ the first voice recognition method that determines a language of input audio data and converts the audio data into text data in the determined language as mentioned above. Moreover, the voice recognition module705may employ the second voice recognition method that converts input audio data to phonemes using acoustic models of a plurality of languages and converts the audio data into text data in each of the languages using the corresponding language model.

Next, the CPU402executes a language determination process on the basis of the text data converted in the step S1101and the language determination result (a step S1102). It should be noted that the contents of the language determination process of the step S1102differ on the basis of the method (the first voice recognition method or the second voice recognition method) used for conversion of the text data in the step S1101. For example, when the first voice recognition method is used for conversion of the text data in the step S1101, the CPU402executes a first language determination process ofFIG. 12mentioned later. In the meantime, when the second voice recognition method is used for conversion of the text data in the step S1101, the CPU402executes a second language determination process ofFIG. 13mentioned later.

Next, the CPU402executes an operation determination process ofFIG. 14mentioned later (a step S1103) and stores the operation information that is the determination result of the type of the user's function call instruction into the RAM403. Next, the CPU402determines whether the operation information stored in the RAM403is “job execution” (a step S1104).

As a result of the determination in the step S1104, when the operation information is “job execution”, the CPU402executes the job execution process ofFIG. 15mentioned later (a step S1105) and finishes the voice-operation-service execution process. As a result of the determination in the step S1104, when the operation information is not “job execution”, the CPU402determines whether operation information is “job setting” (a step S1106).

As a result of the determination in the step S1106, when the operation information is “job setting”, the CPU402executes a job setting process ofFIG. 23mentioned later (a step S1107) and finishes the voice-operation-service execution process. As a result of the determination in the step S1106, when the operation information is not “job setting”, the CPU402generates an operation guide message that is a text message for urging the input of an operation key word (a step S1108). And then, the CPU402stores the operation guide message into the audio data storage area in the RAM403. The operation guide message is “Please give an operation of COPY, EMAILSEND, or the like that you want to execute.”, for example. Next, the CPU402controls the data transmission/reception module701to transmit the operation guide message stored in the RAM403through the network I/F406to the smart speaker102(a step S1109) and finishes the voice-operation-service execution process.

FIG. 12is a flowchart showing procedures of the first language determination process executed in the step S1102when the first voice recognition method is used for conversion of text data in the step S1101inFIG. 11.

As shown inFIG. 12, the CPU402clears a temporary storage area that is a part of the storage area of the RAM403(a step S1201). The temporary storage area is a storage area used in the first language determination process, for example, includes a language-determination-result temporary storage area, a morpheme string storage area, a group ID storage area, and a language-determination-result storage area. Next, the CPU402stores the language determination result of the audio data performed in the voice recognition process in the step S1101into the language-determination-result temporary storage area of the RAM403(a step S1202). Next, the CPU402analyzes the above-mentioned text data by the morphological analysis module706to extract a morpheme string corresponding to the determined language stored in the language-determination-result temporary storage area and converts the morphemes that constitute the morpheme string into group IDs by the group ID determination module707. Next, the CPU402stores the morpheme string in the morpheme string storage area and stores the group IDs in the group ID storage area (a step S1203).

Next, the CPU402obtains the language determination exception flag of each of the morphemes that constitute the above-mentioned morpheme string from the group ID lists711,722, and713inFIG. 7Band the group ID lists721,722, and723inFIG. 7C. The CPU402determines that a morpheme of which the language determination exception flag is “YES” is a determination exception morpheme (language-specification-impossible-word information). The CPU402determines whether all the morphemes that constitute the morpheme string are the determination exception morphemes (a step S1204).

As a result of the determination in the step S1204, when at least one morpheme is not the determination exception morpheme, the CPU402stores the language determination result stored in the language-determination-result temporary storage area into the language-determination-result storage area (a step S1205). The language determination result stored in the language-determination-result temporary storage area is the language determination result of the audio data obtained in the voice recognition process of the step S1101. After that, the language determination process is finished.

As a result of the determination in the step S1204, when all the morphemes are the determination exception morphemes, the CPU402stores “unknown” showing that the language determination is impossible into the language-determination-result storage area (a step S1206). After that, the language determination process is finished.

FIG. 13is a flowchart showing procedures of the second language determination process executed in the step S1102when the second voice recognition method is used for conversion of text data in the step S1101inFIG. 11.

As shown inFIG. 13, the CPU402clears a temporary storage area that is a part of the storage area of the RAM403(a step S1301). The temporary storage area is used in the second language determination process and includes a Japanese-voice-recognition-data storage area, an English-voice-recognition-data storage area, a Japanese morpheme string storage area, a Japanese group ID storage area, an English-morpheme-string storage area, an English group ID storage area, a language-determination-result storage area, and a group ID storage area.

Next, the CPU402stores voice recognition data (hereinafter referred to as “Japanese voice recognition data”) that includes the language setting “Japanese” into the Japanese-voice-recognition-data storage area. The Japanese voice recognition data includes the text data that is obtained because the voice recognition module705applies the voice recognition process to the audio data in Japanese (a step S1302). Moreover, the CPU402stores voice recognition data (hereinafter referred to as “English voice recognition data”) that includes the language setting “English” into the English-voice-recognition-data storage area. The English voice recognition data includes the text data that is obtained because the voice recognition module705applies the voice recognition process to the audio data in English (the step S1302).

Next, the CPU402analyzes the text data included in the Japanese voice recognition data by the morphological analysis module706to extract a morpheme string corresponding to Japanese and converts the morphemes that constitute the morpheme string into group IDs by the group ID determination module707. The CPU402stores the morpheme string (hereinafter referred to as a “Japanese morpheme string”) in the Japanese morpheme string storage area and stores the group IDs (hereinafter referred to as “Japanese group IDs”) in the Japanese group ID storage area (a step S1303).

Next, the CPU402analyzes the text data included in the English voice recognition data by the morphological analysis module706to extract a morpheme string corresponding to English and converts the morphemes that constitute the morpheme string into group IDs by the group ID determination module707. The CPU402stores the morpheme string (hereinafter referred to as an “English morpheme string”) in the English morpheme string storage area and stores the group IDs (hereinafter referred to as “English group IDs”) in the English group ID storage area (a step S1304).

Next, the CPU402determines whether the Japanese group ID storage area is empty (a step S1305). In the step S1305, when no group ID is stored in the Japanese group ID storage area, the CPU402determines that the Japanese group ID storage area is empty. In the meantime, when at least one group ID is stored in the Japanese group ID storage area, the CPU402determines that the Japanese group ID storage area is not empty.

As a result of the determination in the step S1305, when the Japanese group ID storage area is not empty, the CPU402obtains the language determination exception flags of the morphemes that constitutes the Japanese morpheme string from the group ID lists711,712, and713inFIG. 7B. The CPU402determines whether all the morphemes that constitute the Japanese morpheme string are the determination exception morphemes (a step S1306).

As a result of the determination in the step S1306, when at least one morpheme that constitutes the Japanese morpheme string is not the determination exception morpheme, the CPU402stores the group IDs stored in the Japanese group ID storage area to the group ID storage area (a step S1307). It should be noted that a group ID stored in the Japanese group ID storage area is a Japanese group ID. Next, the CPU402stores the language setting “Japanese” in the language-determination-result storage area (a step S1308). After that, the language determination process is finished.

When all the morphemes that constitute the Japanese morpheme string are the determination exception morphemes as a result of the determination in the step S1306or when the Japanese group ID storage area is empty as a result of the determination in the step S1305, the CPU402determines whether the English group ID storage area is empty (a step S1309).

As a result of the determination in the step S1309, when the English group ID storage area is not empty, the CPU402obtains the language determination exception flags of the morphemes that constitutes the English morpheme string from the group ID lists721,722, and723inFIG. 7C. The CPU402determines whether all the morphemes that constitute the English morpheme string are the determination exception morphemes (a step S1310).

As a result of the determination in the step S1310, when at least one morpheme that constitutes the English morpheme string is not the determination exception morpheme, the CPU402stores the group IDs stored in the English group ID storage area to the group ID storage area (a step S1311). It should be noted that a group ID stored in the English group ID storage area is an English group ID. Next, the CPU402stores the language setting “English” in the language-determination-result storage area (a step S1312). After that, the language determination process is finished.

When all the morphemes that constitute the English morpheme string are the determination exception morphemes as a result of the determination in the step S1310or when the English group ID storage area is empty as a result of the determination in the step S1309, the CPU402stores “unknown” showing that the language determination is impossible into the language-determination-result storage area (a step S1313). After that, the language determination process is finished. In this embodiment, when a phrase that a user utters consists of only determination exception morphemes, such as “kopi”, “copy”, “kopisetteingu”, and “copy setting”, as shown inFIG. 30, “unknown” is stored in the language-determination-result storage area in this way. Moreover, when the user utters phrases, such as “yonbukopishite”, “noudosetteiyon”, “four copies”, and “set density four”, that include morphemes other than the determination exception morphemes (other than the language-specification-impossible-word information), “English” or “Japanese” is stored in the language-determination-result storage area.

FIG. 14is a flowchart showing procedures of the operation determination process of the step S1103inFIG. 11.

As shown inFIG. 14, the CPU402determines whether only one group ID (hereinafter referred to as a “job type specifying group ID”) that specifies the job type is stored in the group ID storage area of the RAM403(a step S1401). The job type specifying group IDs are, for example, “FNC00001” corresponding to the job type “COPY” and “FNC00004” corresponding to the job type “EMAILSEND”.

As a result of the determination in the step S1401, when only one job type specifying group ID is stored in the group ID storage area, the CPU402determines whether a group ID (hereinafter referred to as a “setting specification group ID”) that specifies “setting” is stored in the group ID storage area (a step S1402). The setting specification group ID is “FNC00003” corresponding to “setting”, for example.

As a result of the determination in the step S1402, when the setting specification group ID is stored in the group ID storage area, the CPU402stores “job setting” showing that the type of the user's function call instruction is “setting” into the RAM403as the operation information (a step S1403), and the operation determination process is finished.

As a result of the determination in the step S1402, when no setting specification group ID is stored in the group ID storage area, the CPU402stores “job execution” showing that the type of the user's function call instruction is “execution of a job” into the RAM403as the operation information (a step S1404), and the operation determination process is finished.

As a result of the determination in the step S1401, when a plurality of job type specifying group IDs are stored in the group ID storage area or when no job type specifying group ID is stored in the group ID storage area, the CPU402stores “unknown” showing that the type of the user's function call instruction is unknown into the RAM403as the operation information (a step S1405), and the operation determination process is finished.

FIG. 15is a flowchart showing procedures of the job execution process of the step S1105inFIG. 11.

As shown inFIG. 15, the CPU402determines whether essential job setting group IDs are completed in the group ID storage area of the RAM403(a step S1501). An essential job setting group ID is a group ID corresponding to a setting that a user has to set up to execute a job. For example, the essential job setting group ID of the job type “EMAILSEND” is “CNF00004” that shows a destination. The essential job setting group IDs differ depending on the job type. There is a job type that has no essential job setting group ID and there is a job type that has a plurality of essential job setting group IDs.

As a result of the determination in the step S1501, when the essential job setting group IDs are completed in the group ID storage area, the CPU402executes a job information generation process ofFIG. 16mentioned later (a step S1502) to generate the language-set job information that is a device operation data for executing the job by the MFP101. Next, the CPU402transmits the language-set job information concerned to the MFP101through the network I/F406(a step S1503). Next, the CPU402determines whether the job-execution end notification is received from the MFP101(a step S1504). In the embodiment, when the job is completed or when the job is suspended due to occurrence of an error, the MFP101transmits the job-execution end notification including the information that indicates such a job end state to the cloud server103. The CPU402waits until receiving the job-execution end notification from the MFP101. When the job-execution end notification is received from the MFP101(YES in the step S1504), the CPU402generates a job end audio message that is a text message corresponding to the received job-execution end notification (a step S1505). In the step S1505, “Job has been completed” that is a message at a time of normal end or “End due to error” that is a message at a time of occurrence of paper jam or an error in the MFP101are generated, for example.

Next, the CPU402stores the job end audio message concerned in the audio message storage area in the RAM403. Next, the CPU402transmits the audio message stored in the audio message storage area to the smart speaker102through the network I/F406(a step S1506) and finishes the job execution process.

As a result of the determination in the step S1501, when the essential job setting group IDs are not completed in the group ID storage area, the CPU402generates a job-setting-guidance audio message (a step S1507). The job-setting-guidance audio message is a text message to urge the input of settings that is needed for execution of a job. For example, when no destination is designated in a state where the user has designated “EMAILSEND”, the j ob-setting-guidance audio message “Please input a transmission destination” is generated. The CPU402stores the generated job-setting-guidance audio message in the audio message storage area and executes the process of the step S1506.

FIG. 16is a flowchart showing procedures of a job information generation process of the step S1502inFIG. 15.

As shown inFIG. 16, the CPU402clears the temporary storage area used for a job information generation process on the RAM403(a step S1601). This temporary storage area includes a language-determination-character-string storage area, a job character string storage area, and a j ob-setting-character-string storage area. Next, the CPU402parameterizes the language setting (a step S1602). Specifically, the CPU402generates the parameter character string corresponding to the language determination result stored in the language-determination-result storage area in the RAM403. For example, when “Japanese” is stored as the language determination result in the language-determination-result storage area, the CPU402generates, as shown inFIG. 17, the character string {“language”: “Japanese” } showing that the language setting is Japanese and stores the character string concerned in the language-determination-character-string storage area. Moreover, when “English” is stored as the language determination result in the language-determination-result storage area, the CPU402generates, as shown inFIG. 18, the character string {“language”: “English”} showing that the language setting is English and stores the character string concerned in the language-determination-character-string storage area.

Next, the CPU402parameterizes the job type (a step S1603). Specifically, the CPU402extracts the job type specifying group ID from the group ID storage area in the RAM403and generates a parameter character string corresponding to the job type specifying group ID concerned. For example, as shown inFIG. 17orFIG. 18, when “NUM00004”, “CNF00001”, and “FNC00001” are stored in the group ID storage area, the CPU402extracts “FNC00001” that is the job type specifying group ID from among them and generates the character string {“jobName”: “copy” } as the parameter character string corresponding to “FNC00001”. The CPU402stores the generated character string in the job character string storage area.

Next, the CPU402determines whether the stored group ID is the setting specification group ID sequentially from the head address of the group ID storage area (a step S1604).

As a result of the determination in the step S1604, when the stored group ID is the setting specification group ID, the CPU402parameterizes the job setting (a step S1605). Specifically, the CPU402generates a character string corresponding to the group ID determined as the setting specification group ID and stores the character string concerned in the job-setting-character-string storage area in the RAM403. After that, the job information generation process returns to the step S1604. In this way, the character string corresponding to the setting specification group ID stored in the group ID storage area is generated in the embodiment. For example, when “NUM00004”, “CNF00001”, and “FNC00001” are stored in the group ID storage area, the CPU402generates, as shown inFIG. 17andFIG. 18, “copies” as the character string corresponding to “CNF00001” that is the setting specification group ID. Moreover, the CPU402generates “4” as a character string corresponding to “NUM00004” that is the setting specification group ID as shown inFIG. 17orFIG. 18. The CPU402stores these generated character strings in the job-setting-character-string storage area.

As a result of the determination in the step S1604, when the stored group ID is not the setting specification group ID, the CPU402determines whether all the group IDs stored in the group ID storage area have been subjected to the determination of the step S1604(a step S1606).

As a result of the determination in the step S1606, when not all the group IDs stored in the group ID storage area have been subjected to the determination of the step S1604, the job information generation process returns to the step S1604. As a result of the determination in the step S1606, when all the group IDs stored in the group ID storage area have been subjected to the determination of the step S1604, the CPU402generates the language-set job information, which is device operation data for instructing the MFP101to execute a job, on the basis of the character strings stored in the language-determination-character-string storage area, the job character string storage area, and the job-setting-character-string storage area (a step S1607). The language-set job information is, for example, the data in the JSON format as shown inFIG. 8. The data format of the language-set job information is not restricted to the JSON format. The data format may be another format like the XML format. After that, the job information generation process is finished.

FIG. 19is a flowchart showing a language setting switching process executed by the MFP101that receives the language-set job information from the cloud server103. The language setting switching process ofFIG. 19is achieved because the CPU202of the MFP101runs the control program developed from the ROM204to the RAM203.

As shown inFIG. 19, the CPU202obtains the language setting from the received language-set job information by the data analysis module502and determines whether the obtained language setting is “unknown” (a step S1901).

As a result of the determination in the step S1901, when the obtained language setting is “unknown”, the language setting switching process proceeds to the step S1903mentioned later. As a result of the determination in the step S1901, when the obtained language setting is not “unknown”, the CPU202updates the display language of the operation panel209(a step S1902). Specifically, the CPU202stores the obtained language setting in an MFP-language-setting storage area in the storage unit205. Next, the CPU202obtains the job type and job setting from the received language-set job information. The CPU202generates job primary information corresponding to the obtained job type (a step S1903) and stores the job primary information concerned in the RAM203. Moreover, the CPU202sets a parameter corresponding to the obtained job setting to the above-mentioned job primary information.

Next, the CPU202determines whether the job is executable (a step S1904). In the step S1904, the CPU202determines that the job is not executable when the MFP101cannot execute new job because of execution of another job or occurrence of an error, for example. In the meantime, when the MFP101can execute a new job, the CPU202is determined that a job is executable.

As a result of the determination in the step S1904, when a job is not executable, the language setting switching process proceeds to a step S1907mentioned later. As a result of the determination in the step S1904, when a job is executable, the CPU202transmits a job-execution start notification to the cloud server103through the network104by the data transmission/reception module501(a step S1905). Next, the CPU202executes a job on the basis of the job primary information generated in the step S1903(a step S1906). Next, the CPU202transmits the job-execution end notification to the cloud server103through the network104by the data transmission/reception module501(a step S1907). The job-execution end notification includes a job execution result. For example, when the job executed in the step S1906is completed normally, the job-execution end notification includes the job execution result showing that the job has been completed normally. Moreover, when it is determined that a job is not executable in the step S1904or when the job executed in the step S1906has been finished abnormally due to a paper jam etc., the job-execution end notification includes the job execution result that shows an error. The cloud server103generates an audio message corresponding to the job execution result included in the received job-execution end notification. After the CPU202executes the process of the step S1907, the language setting switching process is finished.

FIG. 20is a view showing screen transitions of the operation panel209of the MFP101when execution of a copy job is instructed by a voice input.

When the MFP101receives the language-set job information including {“language”: “Japanese”}, {“operation”: “jobStart”}, and {“jobName”: “copy”} from the cloud server103in a state where the home screen2001is displayed on the operation panel209as shown inFIG. 20, the MFP101sets the language setting to Japanese and starts execution of the copy job. When the copy job is executed in the state where the language setting is set to Japanese, a copying screen2002the display language of which is Japanese is displayed on the operation panel209.

Moreover, when the MFP101receives the language-set job information including {“language”: “English”}, {“operation”: “jobStart”}, and {“jobName”: “copy”} from the cloud server103in the state where the home screen2001is displayed on the operation panel209, the MFP101sets the language setting to English and starts execution of the copy job. When the copy job is executed in the state where the language setting is set to English, a copying screen2003the display language of which is English is displayed on the operation panel209.

FIG. 21is a view showing screen transitions of the operation panel209of the MFP101when execution of an EMAILSEND job is instructed by a voice input.

When the MFP101receives the language-set job information including {“language”: “Japanese”}, {“operation”: “jobStart”}, and {“jobName”: “emailSend”} from the cloud server103in the state where the home screen2001is displayed on the operation panel209as shown inFIG. 21, the MFP101sets the language setting to Japanese and starts execution of the EMAILSEND job. When the EMAILSEND job is executed in the state where the language setting is set to Japanese, a scanning screen2101the display language of which is Japanese is displayed on the operation panel209.

Moreover, when the MFP101receives the language-set job information including {“language”: “English”}, {“operation”: “jobStart”}, and {“jobName”: “emailSend”} from the cloud server103in the state where the home screen2001is displayed on the operation panel209, the MFP101sets the language setting to English and starts execution of the EMAILSEND job. When the EMAILSEND job is executed in the state where the language setting is set to English, a scanning screen2102the display language of which is English is displayed on the operation panel209. Although the home screen2001is described as one example of a job executable screen in the embodiment, the job executable screen is not limited to the home screen2001. Moreover, when the MFP101receives the language-set job information from the cloud server103in the power saving mode in which the job executable screen is not displayed and the powers of the operation panel209and print engine211are OFF, the MFP101may set up the language setting on the basis of the received language-set job information and may execute the job as mentioned above.

FIG. 22is a sequence chart showing procedures of a process executed when the image forming system100ofFIG. 1receives a job setting change instruction by a voice input. It should be noted that the smart speaker102, MFP101, and cloud server103shall be communicable mutually inFIG. 22as with the description aboutFIG. 10. Moreover, the process ofFIG. 22assumes that a home screen2801inFIG. 28on which functions, such as copy, scan, and print, can be called are displayed on the operation panel209of the MFP101.

In a step S2201inFIG. 22, the user gives an instruction to the smart speaker102to start a voice operation as with the step S1001.

When the start instruction of the voice operation is detected, the display module606of the audio control module600in the smart speaker102lights the LED312as a notification to show being in the utterance processing state in a step S2202like the step S1002. Moreover, the process by the voice obtainment module604is started.

In a step S2203, the user performs the function call instruction to the smart speaker102. For example, the user utters a phrase, such as “kopinoudosetteiyon” or “Set copy density four”, that is the job setting change instruction as a function call instruction following the wake word detected in the step S2201. Audio data is generated on the basis of user's voice that is obtained by the voice obtainment module604. When the pause period of the predetermined period continues, the utterance end determination module608determines that the utterance ends.

In a step S2204, the display module606of the audio control module600blinks the LED312as a notification to indicate being in the response processing state depending on the utterance end determination as with the step S1004. Moreover, the process by the voice obtainment module604is completed. In a step S2205, the data transmission/reception module601transmits the generated audio data to the cloud server103as with the step S1005.

In a step S2206, the audio data conversion control module700in the cloud server103executes the voice-operation-service execution process ofFIG. 11mentioned above. In the voice-operation-service execution process, the language-set job information mentioned later is transmitted to the MFP101, for example.

In a step S2207, the device control module500in the MFP101executes a language setting switching process ofFIG. 19mentioned later on the basis of the job setting information received from the cloud server103.

In a step S2208, the data transmission/reception module601in the smart speaker102receives an audio message from the cloud server103. In the next step S2209, the audio reproduction module605reproduces the composite audio data into which the audio message received in the step S2208is converted. For example, the audio reproduction module605reproduces the composite audio data “density setting will be started” through the loudspeaker310.

In a step S2210, the data transmission/reception module601receives an audio message that is different from the audio message received in the step S2208from the cloud server103. Moreover, the data transmission/reception module601receives the interactive session end notification that finishes then interactive session with the user from the cloud server103.

In a step S2211, the audio reproduction module605reproduces the composite audio data into which the audio message received in the step S2210is converted. For example, the audio reproduction module605reproduces the composite audio data “density setting has been finished” through the loudspeaker310.

In a step S2212, the display module606turns off the LED312as a notification showing that the smart speaker102is in the standby state in response to the reception of the interactive session end notification by the data transmission/reception module601in the step S2210.

In a step S2213, the audio control module600shifts the smart speaker102to the standby state in response to the reception of the interactive session end notification by the data transmission/reception module601in the step S2210.

FIG. 23is a flowchart showing procedures of the job setting process of the step S1107inFIG. 11. The job setting process ofFIG. 23is executed when the cloud server103receives audio data generated on the basis of user's voice that is a job setting change instruction from the smart speaker102.

As shown inFIG. 23, the CPU402generates language-set job information including a set value used when the MFP101executes a job by executing a job-setting-information generation process ofFIG. 24mentioned later (a step S2301). Next, the CPU402transmits the language-set job information concerned to the MFP101through the network I/F406(a step S2302). Next, the CPU402determines whether a job setting end notification is received from the MFP101(a step S2303). In the embodiment, when the job setting is completed normally or when the job setting is suspended due to occurrence of an error, the MFP101transmits the job setting end notification including the information that indicates such a job setting end state to the cloud server103. The CPU402waits until receiving the job setting end notification from the MFP101. When the job setting end notification is received from the MFP101(YES in the step S2303), the CPU402generates a job setting end audio message that is a text message corresponding to the received job setting end notification (a step S2304). In the step S2304, the CPU402generates “Job setting has been completed” that is a message at a time of normal end or “Job setting has been impossible” that is a message at a time of occurrence of paper jam or an error in the MFP101, for example.

Next, the CPU402stores the job setting end audio message in the audio message storage area in the RAM403. Next, the CPU402transmits the audio message stored in the audio message storage area to the smart speaker102through the network I/F406(a step S2305) and finishes the job setting process.

FIG. 24is a flowchart showing procedures of the job setting information generation process of the step S2301inFIG. 23.

As shown inFIG. 24, the CPU402clears the temporary storage area used for the job setting information generation process on the RAM403(a step S2401). The temporary storage area includes a language-determination-character-string storage area, a job character string storage area, and a j ob-setting-character-string storage area. Next, the CPU402parameterizes the language setting (a step S2402). Specifically, the CPU402generates the parameter character string corresponding to the language determination result stored in the language-determination-result storage area in the RAM403. For example, when “Japanese” is stored as the language determination result in the language-determination-result storage area, the CPU402generates, as shown inFIG. 25, the character string {“language”: “Japanese” } showing that the language setting is Japanese and stores the character string concerned in the language-determination-character-string storage area. Moreover, when “English” is stored as the language determination result in the language-determination-result storage area, the CPU402generates, as shown inFIG. 26, the character string {“language”: “English”} showing that the language setting is English and stores the character string concerned in the language-determination-character-string storage area.

Next, the CPU402parameterizes the job type (a step S2403). Specifically, the CPU402extracts the job type specifying group ID from the group ID storage area in the RAM403and generates a parameter character string corresponding to the job type specifying group ID concerned. For example, as shown inFIG. 25orFIG. 26, when “FNC00001”, “NUM00004”, “CNF00002”, and “FNC00003” are stored in the group ID storage area, the CPU402extracts “FNC00001” that is the job type specifying group ID from among them and generates the character string {“jobName”: “copy” } as the parameter character string corresponding to “FNC00001”. The CPU402stores the generated character string in the job character string storage area.

Next, the CPU402determines whether the stored group ID is the setting specification group ID sequentially from the head address of the group ID storage area (a step S2404).

As a result of the determination in the step S2404, when the stored group ID is the setting specification group ID, the CPU402parameterizes the job setting (a step S2405). Specifically, the CPU402generates a character string corresponding to the group ID determined as the setting specification group ID and stores the character string concerned in the job-setting-character-string storage area in the RAM403. After that, the job setting information generation process returns to the step S2404. In this way, the character string corresponding to the setting specification group ID stored in the group ID storage area is generated in the embodiment. For example, when “FNC00001”, “NUM00004”, “CNF00002”, and “FNC00003” are stored in the group ID storage area, the CPU402generates, as shown inFIG. 25andFIG. 26, “density” as the character string corresponding to “CNF00002” that is the setting specification group ID. Moreover, the CPU402generates “4” as a character string corresponding to “NUM00004” that is the setting specification group ID as shown inFIG. 25orFIG. 26. The CPU402stores these generated character strings in the job-setting-character-string storage area.

As a result of the determination in the step S2404, when the stored group ID is not the setting specification group ID, the CPU402determines whether all the group IDs stored in the group ID storage area have been subjected to the determination of the step S2404(a step S2406).

As a result of the determination in the step S2406, when not all the group IDs stored in the group ID storage area have been subjected to the determination of the step S2404, the job setting information generation process returns to the step S2404. As a result of the determination in the step S2406, when all the group IDs stored in the group ID storage area have been subjected to the determination of the step S2404, the CPU402generates the language-set job information, which is device operation data for instructing the MFP101to execute a job, on the basis of the character strings stored in the language-determination-character-string storage area, the job character string storage area, and the job-setting-character-string storage area (a step S2407). The language-set job information is, for example, the data in the JSON format as shown inFIG. 9. The data format of the language-set job information is not restricted to the JSON format. The data format may be another format like the XML format.

FIG. 27is a flowchart showing a language setting switching process executed by the MFP101that receives the language-set job information from the cloud server103. The language setting switching process ofFIG. 27is achieved because the CPU202of the MFP101runs the control program developed from the ROM204to the RAM203.

As shown inFIG. 27, the CPU202obtains the language setting from the received language-set job information by the data analysis module502and determines whether the obtained language setting is “unknown” (a step S2701).

As a result of the determination in the step S2701, when the obtained language setting is “unknown”, the language setting switching process proceeds to a step S2703mentioned later. As a result of the determination in the step S2701, when the obtained language setting is not “unknown”, the CPU202updates the display language of the operation panel209(a step S2702). Specifically, the CPU202stores the obtained language setting in the MFP-language-setting storage area in the storage unit205. Next, the CPU202obtains the job type and job setting from the received language-set job setting information. The CPU202generates job primary information corresponding to the obtained job type (a step S2703) and stores the job primary information concerned in the RAM203. Moreover, the CPU202sets a parameter corresponding to the obtained job setting to the above-mentioned job primary information.

Next, the CPU202determines whether transition of a screen of the operation panel209to a j ob setting screen is possible (a step S2704). A user can set a set value, which is needed to execute a job, on the job setting screen. In the step S2704, when the transition of the screen of the operation panel209is impossible because the MFP101is executing another job or causes an error, the CPU202determines that the transition of the screen of the operation panel209to the job setting screen is impossible, for example. In the meantime, when the transition of the screen of the operation panel209is possible, the CPU202determines that the transition of the screen of the operation panel209to the job setting screen is possible.

As a result of the determination in the step S2704, when the transition of the screen of the operation panel209to the job setting screen is impossible, the language setting switching process proceeds to a step S2706mentioned later. As a result of the determination in the step S2704, when the transition of the screen of the operation panel209to the job setting screen is possible, the job setting screen is displayed on the operation panel209(a step S2705). Next, the CPU202transmits the job setting end notification to the cloud server103through the network104by the data transmission/reception module501(a step S2706). The job setting end notification includes a job setting result. For example, when the screen transition is completed normally, the job setting end notification includes the job setting result showing that the screen transition has been completed normally. Moreover, when it is determined that the transition of the screen of the operation panel209to the job setting screen is impossible in the step S2704, the job setting end notification includes the job setting result indicating an error. After the CPU202executes the process of the step S2706, the language setting switching process is finished.

FIG. 28is a view showing screen transitions of the operation panel209of the MFP101when the setting of a copy job is input by user's voice.

As shown inFIG. 28, when the MFP101receives the language-set job information including {“language”: “Japanese”}, {“operation”: “jobSetting”}, and {“jobName”: “copy”} from the cloud server103in the state where the home screen2801is displayed on the operation panel209, the MFP101sets the language setting to Japanese. A copy setting screen2802the display language of which is Japanese is displayed on the operation panel209. After that, when the user gives a job execution instruction from the smart speaker102or the operation panel209, the job control module503executes the copy job, and a copy executing screen2803the display language of which is Japanese is displayed on the operation panel209.

Moreover, when the MFP101receives the language-set job information including {“language”: “English”}, {“operation”: “jobSetting”}, and {“jobName”: “copy”} from the cloud server103in the state where the home screen2801is displayed on the operation panel209, the MFP101sets the language setting to English. A copy setting screen2804the display language of which is English is displayed on the operation panel209. After that, when the user gives a job execution instruction from the smart speaker102or the operation panel209, the job control module503executes the copy job, and a copy executing screen2805the display language of which is English is displayed on the operation panel209.

FIG. 29is a view showing screen transitions of the operation panel209of the MFP101when the setting of an EMAIL SEND job is input by user's voice.

As shown inFIG. 29, when the MFP101receives the language-set job information including {“language”: “Japanese”}, {“operation”: “jobSetting”}, and {“jobName”: “emailSend”} from the cloud server103in the state where the home screen2801is displayed on the operation panel209, the MFP101sets the language setting to Japanese. A scan setting screen2901the display language of which is Japanese is displayed on the operation panel209. After that, when the user gives a job execution instruction from the smart speaker102or the operation panel209, the job control module503executes the EMAILSEND job, and a scanning screen2902the display language of which is Japanese is displayed on the operation panel209.

Moreover, when the MFP101receives the language-set job information including {“language”: “English”}, {“operation”: “jobSetting”}, and {“jobName”: “emailSend”} from the cloud server103in the state where the home screen2801is displayed on the operation panel209, the MFP101sets the language setting to English. A scan setting screen2903the display language of which is English is displayed on the operation panel209. After that, when the user gives a job execution instruction from the smart speaker102or the operation panel209, the job control module503executes the EMAILSEND job, and a scanning screen2904the display language of which is English is displayed on the operation panel209.

According to the embodiment mentioned above, a morpheme string that consists of a plurality of morphemes is obtained on the basis of a phrase that the smart speaker102obtained, a language is specified using the morpheme string concerned, and a display language of the operation panel209is updated on the basis of the specified language. That is, the display language of the operation panel209is changed to a user's using language without giving a setting operation about the display language of the operation panel every time when the user starts to use the MFP101. This can reduce the time and effort of a user for setting a display language.

Moreover, in the embodiment mentioned above, the cloud server103obtains a morpheme string on the basis of user's voice that the smart speaker102obtained. Accordingly, the cloud server103can promptly execute the process to specify a language using the obtained morpheme string.

In the embodiment mentioned above, when a determination exception morpheme is included in a morpheme string, a language is specified using morphemes other than the determination exception morpheme in the morpheme string. Thereby, accuracy of specifying a language is improved.

Although the present invention is described using the embodiment mentioned above, the present invention is not limited to the embodiment mentioned above. For example, when the MFP101is provided with the microphone308, the MFP101may transmit audio data generated on the basis of user's voice obtained with the microphone308to the cloud server103.

The MFP101may be configured to be connectable to another external display device without having the operation panel209.

Moreover, when the external display device is provided with the microphone308, the external display device may transmit audio data generated on the basis of user's voice obtained with the microphone308to the cloud server103.

Furthermore, when the MFP101that is connectable to an external display device without having the operation panel209is provided with the microphone308, the MFP101may transmit audio data generated on the basis of user's voice obtained with the microphone308to the cloud server103.

The MFP101may obtain a morpheme string on the basis of audio data generated on the basis of user's voice obtained with the microphone308of the MFP101or audio data obtained from the smart speaker102and may transmit the obtained morpheme string to the cloud server103. This can distribute a load needed to execute the process to obtain the morpheme string on the cloud server103.

When only the determination exception morphemes are included in the morpheme string obtained on the basis of the obtained phrase, the display language of the operation panel209may be updated on the basis of a language specified using a morpheme string obtained from another phrase that the smart speaker102obtains after obtaining the previous phrase. For example, even if the language determination is impossible because the phrase uttered by the user consists of only determination exception morphemes like “kopi”, the display language of the operation panel209will be changed to Japanese when the user will utter a phrase, such as “sanbukopishite”, including morphemes other than the determination exception morphemes and the language will be specified as Japanese. Thereby, the display language can be switched at a timing of specifying the language during the interactive session without requiring a user to set the display language.

Other Embodiments

This application claims the benefit of Japanese Patent Application No. 2020-084478, filed May 13, 2020, which is hereby incorporated by reference herein in its entirety.