Patent Description:
In recent years, the development of deep learning enables accurate recognition of nonverbal information such as a person's line of sight and facial expression from a video image in real time. This technology is applied to various applications such as automatic analysis of surveillance camera images and health condition monitoring. Further, in recent years, a nonverbal information conversion technology developed in conjunction with the nonverbal information recognition technology is attracting attention. These techniques enable to give a desired impression to a partner in a conversation using a video call, for example.

To express and recognize nonverbal information has some significance in dialogue communication. On the other hand, issues may occur in dialogue due to the inability to handle nonverbal information correctly or the misinterpretation of nonverbal information from one to another. To address such issues, for example, <CIT> discloses estimating emotion from voice, generating character image that helps a conversation partner to recognize the emotion, and displaying the generated character image.

Since there are differences in recognition and expression of nonverbal information between individuals and between cultures, estimation and expression are to be performed according to individuals. However, according to the conventional method, there is room for improvement in converting nonverbal information in consideration of individuality in expression and recognition of nonverbal information. <CIT> discloses a message storage server that performs emotion recognition on a message by speech or video transmitted from a mobile communication terminal. <CIT> discloses a processor that receives, from a first terminal, video data in which a first user appears, analyzes the video data, selects a movement pattern corresponding to a nonverbal behavior of the first user from an avatar movement pattern group, and transmits control data indicating the selected movement pattern to a second terminal in order to move an avatar corresponding to the first user in a virtual space displayed on the second terminal, on the basis of the selected movement pattern. <CIT> discloses a terminal supporting an avatar base method for video communications.

In one aspect of the present invention, an information processing apparatus converts nonverbal information. The information processing apparatus includes intention estimation means for estimating an intention indicated in first nonverbal information, which is nonverbal information of a first user, based on a feature amount of the first nonverbal information and a nonverbal expression model indicating a relation between the first nonverbal information and an intention. The information processing apparatus includes nonverbal information conversion means for converting the first nonverbal information into second nonverbal information to be output to a second user based on the estimated intention and a set conversion rule of the nonverbal information.

In another aspect of the present invention, a nonverbal information conversion system converts nonverbal information. The nonverbal information conversion system includes the information processing apparatus of the above aspect; and a transmission apparatus used by the first user.

In another aspect of the present invention, an information processing method is performed by an information processing apparatus that converts nonverbal information. The information processing method includes estimating an intention indicated in first nonverbal information, which is nonverbal information of a first user, based on a feature amount of the first nonverbal information and a nonverbal expression model indicating a relation between the first nonverbal information and an intention. The information processing method includes converting the first nonverbal information into second nonverbal information to be output to a second user based on the estimated intention and a set conversion rule of the nonverbal information.

In another aspect of the present invention, carrier means carrying computer readable codes for controlling a computer to carry out an information processing method is provided. The information processing method includes estimating an intention indicated in first nonverbal information, which is nonverbal information of a first user, based on a feature amount of the first nonverbal information and a nonverbal expression model indicating a relation between the first nonverbal information and an intention. The information processing method includes converting the first nonverbal information into second nonverbal information to be output to a second user based on the estimated intention and a set conversion rule of the nonverbal information.

According to one aspect of the present invention, in dialogue communication, nonverbal information is converted so that one's intention to be communicated to the other party of the dialogue is intelligibly communicated.

Embodiments of the present invention are described with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions thereof are omitted.

Referring to <FIG>, an overview of a nonverbal information conversion system <NUM> is described according to an embodiment. <FIG> is a schematic diagram illustrating an example of an overall configuration of the nonverbal information conversion system <NUM>. The nonverbal information conversion system <NUM> illustrated in <FIG> is a system that converts nonverbal information exchanged in dialogue communication using video communication, for example.

As illustrated in <FIG>, the nonverbal information conversion system <NUM> includes an information processing apparatus <NUM> that converts nonverbal information, a transmission apparatus <NUM> used by a sender of the nonverbal information, and a reception apparatus <NUM> used by a recipient of the nonverbal information. The information processing apparatus <NUM>, the transmission apparatus <NUM>, and the reception apparatus <NUM> included in the nonverbal information conversion system <NUM> are communicable with each other through a communication network. The communication network includes the Internet, a mobile communication network, and a local area network (LAN), for example. In another example, the communication network includes a network by not only wired communications but also a network by wireless communications such as 3rd generation (<NUM>), 4th generation (<NUM>), 5th generation (<NUM>), Wireless Fidelity® (Wi-Fi), Worldwide Interoperability for Microwave Access (WiMAX) or Long Term Evolution (LTE).

The information processing apparatus <NUM> is a computer that converts nonverbal information so that an intention of a first user, who is the sender, is intelligibly transmitted to a second user, who is the recipient. The information processing apparatus <NUM> acquires data including nonverbal information of the sender, converts the nonverbal information so that an intention of the sender is intelligibly communicated to the recipient, and outputs conversion data obtained by converting the nonverbal information to the acquired data.

In the disclosure, the nonverbal information includes a feature amount such as a user's line of sight, a user's facial expression, a posture of a user's upper limb, a shape of a user's hand, a shape or a posture of a user's arm or foot, or a tone or intonation of user's voice. The intention of the sender includes one or more of a condition of the sender (e.g., pleasant, concentrated, or active), an emotion of the sender (e.g., happy, angry, sad, pleasure, composed, or disgusted), and will of the sender (e.g., instruct, deny, or request) that the sender desires to communicate to the recipient.

The nonverbal information conversion system <NUM> further includes a nonverbal expression model and a nonverbal recognition model constructed for each user in order to improve the conversion accuracy of the nonverbal information. The nonverbal expression model stores a relation between a user's nonverbal expression and an intention. The nonverbal expression model is a model that learns individuality in the nonverbal expression of the user. The nonverbal recognition model stores a relation between a user's nonverbal recognition and an expression. The nonverbal recognition model is a model for learns individuality in nonverbal recognition of the user.

The nonverbal information conversion system <NUM> uses the nonverbal expression model and the nonverbal recognition model learned for each user to convert the nonverbal information, thereby converting the nonverbal information in consideration of individuality in expression and recognition of the nonverbal information. For example, as illustrated in <FIG>, the information processing apparatus <NUM> reads the nonverbal expression model of the sender and the nonverbal recognition model of the recipient, to convert the acquired data including the nonverbal information of the sender into nonverbal information that is easy for the recipient to recognize.

In one example, the information processing apparatus <NUM> is implemented by a single computer. In another example, the information processing apparatus <NUM> is implemented by a plurality of computers. In one example, the information processing apparatus <NUM> is implemented by a computer residing in a cloud environment. In another example, the information processing apparatus <NUM> is implemented by a computer existing residing in an onpremises environment.

The transmission apparatus <NUM> is a computer such as a laptop computer used by the sender in dialogue communication. The reception apparatus <NUM> is a computer such as a laptop computer used by the recipient in dialog communication. The transmission apparatus <NUM> transmits, to the information processing apparatus <NUM>, video data obtained by capturing the sender from the front, for example. The reception apparatus <NUM> controls a display to display video in which the sender appears, based on video data (conversion data) converted by the information processing apparatus <NUM>. The laptop computer is merely one example of each of the transmission apparatus <NUM> and the reception apparatus <NUM>. In another example, each of the transmission apparatus <NUM> and the reception apparatus <NUM> is implemented by a smartphone, a tablet terminal, a wearable terminal, or desktop personal computer (PC). Although <FIG> illustrates an example of dialogue communication is performed between two persons, the nonverbal information conversion system <NUM> is also applicable to dialogue communication between three or more persons. In still another example in dialogue communication, each of the sender and the recipient serves as both the sender and the recipient.

In conversion of nonverbal information in dialogue communication, a technology is known that converts facial expression of a user photographed by a camera or an attribute of the user such as sex or age in real time, or adjusts intonation and tone of machine voice in order to change an impression given to the other party of the dialogue communication. The nonverbal information has a stronger influence on the impression than verbal information. Accordingly, it is considered that a desired impression is given to the other party by using of the technology as described above.

Further, in dialogue communication, expressing and recognizing nonverbal information correctly has some significance. Issues sometimes occur in dialogue due to a case in which nonverbal information is not correctly handled or a difference in interpretation of nonverbal information between the sender and recipient of the nonverbal information. In particular, in the field of nursing care or nursing education, trouble is likely to occur between a person who is not able to handle nonverbal information well and a staff member. In such a situation, there is often a discrepancy between an intention to be communicated and nonverbal expression, or there is often a difference between nonverbal expression and a recognition rule between such the person who is not able to handle nonverbal information well and the staff member, for example. In such a situation, for example, a technology is known that, in response to an input of voice by a sender who does not master sign language expression, syntheses a sign language image that is easy to understand to a recipient. Further, a technology is known that converts visual information into easy-to-recognize information for a visually impaired person or a person with low vision.

However, there are differences in recognition and expression of nonverbal information between individuals and between cultures. To address such an issue, estimation and expression are to be performed according to individuals. According to the conventional method as described above, there is room for improvement in converting nonverbal information in consideration of individuality in expression and recognition of nonverbal information.

To address such an issue, in the nonverbal information conversion system <NUM>, the information processing apparatus <NUM> estimates an intention of the sender based on the nonverbal expression model of the sender by using video data indicating nonverbal information of the sender as an input. Further, the information processing apparatus <NUM> converts the nonverbal information of the sender based on the estimated intention and a set conversion rule namely, a conversion rule based on the nonverbal recognition model of the recipient. Furthermore, the information processing apparatus <NUM> outputs video data, which is the conversion data, indicating the converted nonverbal information, to the recipient. With this configuration, the nonverbal information conversion system <NUM> intelligibly communicate the intention of the sender to the recipient in the dialogue between the sender and the recipient.

Referring to <FIG>, an example of a hardware configuration of each of the apparatuses of the nonverbal information conversion system <NUM> is described, according to an embodiment. The apparatuses of the nonverbal information conversion system <NUM> have a configuration of a general-purpose computer. An example of a hardware configuration of the general-purpose computer is described.

<FIG> is a block diagram illustrating an example of a hardware configuration of a computer. In one example, the computer in the present embodiment and embodiment described below have the same configuration as illustrated in <FIG>. In another example, one or more components or elements are added or deleted as needed. The computer includes a central processing unit (CPU) <NUM>, a read only memory (ROM) <NUM>, a random access memory (RAM) <NUM>, a hard disk (HD) <NUM>, a hard disk drive (HDD) controller <NUM>, a display <NUM>, an external device connection interface (I/F) <NUM>, a communication I/F <NUM>, a bus line <NUM>, a keyboard <NUM>, a pointing device <NUM>, an audio input/output I/F <NUM>, a microphone <NUM>, a speaker <NUM>, a camera <NUM>, a digital versatile disk rewritable (DVD-RW) drive <NUM> and a medium I/F <NUM>.

The CPU <NUM> controls overall operation of the computer. The ROM <NUM> stores a program such as an initial program loader (IPL) to boot the CPU <NUM>. The RAM <NUM> is used as a work area for the CPU <NUM>. The HD <NUM> stores various data such as a program. The HDD controller <NUM> controls reading or writing of various data from or to the HD <NUM> under control of the CPU <NUM>. The display <NUM> is an example of a display device (display means) that displays various types of information such as a cursor, a menu, a window, characters, or an image. In one example, the display <NUM> is a touch panel display provided with an input device (input means). The external device connection I/F <NUM> is an interface that connects the computer to various extraneous sources. The communication I/F <NUM> is an interface for data transmission and reception with other computers or electronic devices. The communication I/F <NUM> is, for example, a communication interface such as a wired or wireless LAN. In another example, the communication I/F <NUM> includes a communication interface for mobile communication such as <NUM>, <NUM>, <NUM>, or LTE, Wi-Fi®, or WiMAX. The bus line <NUM> is, for example, an address bus or a data bus, which electrically connects the elements such as the CPU <NUM> illustrated in <FIG>.

The keyboard <NUM> is an example of an input device (input means) including a plurality of keys for inputting characters, numerical values, various instructions, and the like. The pointing device <NUM> is an example of an input device (input means) that allows a user to select or execute a specific instruction, select an object for processing, or move a cursor being displayed. The keyboard <NUM><NUM> and the pointing device <NUM> are merely examples of the input device (input means). In another example, a touch panel, a voice input device, or the like is used as the input device (input means). In still another example, instead of or in alternative to the display device (display means) such as the display <NUM> and the input device (input means) such as the keyboard <NUM> and the pointing device <NUM>, a user interface (UI) external to the computer is used. The audio input/output I/F <NUM> is a circuit for inputting or outputting an audio signal between the microphone <NUM> and the speaker <NUM> under control of the CPU <NUM>. The microphone <NUM> is an example of an audio collecting device (audio collecting means), which is a built-in type, that receives an input of audio. The speaker <NUM> is an example of an output device (output means), which is a built-in type, that outputs an audio signal. The camera <NUM> is an example of an image capturing device (image capturing means), which is a built-in type, that captures an image of an object to obtain image data. In another example, each of the microphone <NUM>, the speaker <NUM>, and the camera <NUM> is an external device in alternative to the built-in device of the computer. The DVD-RW drive <NUM> controls reading or writing of various data to or from a DVD-RW <NUM>, which is an example of a removable storage medium. In another example, the removable storage medium includes at least one of digital versatile disk-recordable (DVD-R) or a Blu-ray® disc, in addition to or in alternative to the DVD-RW. The medium I/F <NUM> controls reading or writing (storing) of data from or to a storage medium <NUM> such as a flash memory.

For example, any one of the above-described control programs is recorded in a file in a format installable or executable on a computer-readable storage medium for distribution. Examples of the storage medium include, but are not limited to, a compact disc-recordable (CD-R), a digital versatile disk (DVD), a Blu-ray® disc, a secure digital (SD) card, and a universal serial bus (USB) memory. In addition, such storage medium may be provided in the form of a program product to users within a certain country or outside that country. For example, the information processing apparatus <NUM> implements an information processing method according to the present invention by executing a program according to the present invention.

Referring to <FIG>, a functional configuration of the nonverbal information conversion system <NUM> is described according to an embodiment. <FIG> is a block diagram illustrating an example of a functional configuration of the information processing apparatus <NUM>. The information processing apparatus <NUM> includes a data acquisition unit <NUM>, a nonverbal information processing unit <NUM>, and a data output unit <NUM>. These units are functions or means implemented by or that are caused to function by operating any of the hardware elements illustrated in <FIG> in cooperation with instructions of the CPU <NUM> according to the program for the information processing apparatus expanded to the RAM <NUM>.

The data acquisition unit <NUM> is implemented mainly by the communication I/F <NUM> or the external device connection I/F <NUM> operating under control of the CPU <NUM>. The data acquisition unit <NUM> acquires various data transmitted from the transmission apparatus <NUM>.

The nonverbal information processing unit <NUM>, which is implemented mainly by instructions of the CPU <NUM>, converts nonverbal information indicated by data acquired by the data acquisition unit <NUM>.

The data output unit <NUM> is implemented mainly by the communication I/F <NUM> or the external device connection I/F <NUM> operating under control of the CPU <NUM>. The data output unit <NUM> outputs various data obtained by processing by the nonverbal information processing unit <NUM> to the reception apparatus <NUM>.

A detailed description is now given of a configuration of the nonverbal information processing unit <NUM>. The nonverbal information processing unit <NUM> includes a feature extraction unit <NUM>, an intention estimation unit <NUM>, a feature conversion unit <NUM>, a conversion rule setting unit <NUM>, a video conversion unit <NUM>, a model learning unit <NUM>, a data storage unit <NUM>, an intention storage unit <NUM>, a synthesis information storage unit <NUM>, and a storing/reading unit <NUM>. The nonverbal information processing unit <NUM> further includes a storage unit <NUM>, which is implemented by the ROM <NUM>, the HD <NUM>, or the storage medium <NUM> illustrated in <FIG>.

The feature extraction unit <NUM> uses video of a certain person as an input, to extract a feature amount of nonverbal information indicated in the video.

The intention estimation unit <NUM> estimates an intention of the sender based on the nonverbal information of the sender indicated in the data acquired by the data acquisition unit <NUM> and the nonverbal expression model of the sender.

Based on the intention estimated by the intention estimation unit <NUM> and a conversion rule set by the conversion rule setting unit <NUM>, the feature conversion unit <NUM> converts the feature amount of the nonverbal information of the sender indicated in the data acquired by the data acquisition unit <NUM>.

The conversion rule setting unit <NUM> sets the conversion rule for converting the feature amount of the nonverbal information of the sender.

The video conversion unit <NUM> converts the video data acquired by the data acquisition unit <NUM> based on the feature amount converted by the feature conversion unit <NUM>.

The model learning unit <NUM> learns various learning models (a nonverbal expression model <NUM>, a nonverbal recognition model <NUM>, a feature extraction model <NUM>, and a conversion model <NUM>) used for conversion of nonverbal information.

The data storage unit <NUM> stores various data used for conversion of nonverbal information. The intention storage unit <NUM> stores the intention of the sender estimated by the intention estimation unit <NUM>. The synthesis information storage unit <NUM> stores synthesis information corresponding to the video of the sender, which is the conversion data obtained by the conversion by the video conversion unit <NUM>.

The storing/reading unit <NUM> stores various data (or information) in the storage unit <NUM> and/or reads various data (or information) from the storage unit <NUM>. The storage unit <NUM> stores the nonverbal expression model <NUM>, the nonverbal recognition model <NUM>, the feature extraction model <NUM>, and the conversion model <NUM>, which are constructed by conversion processing on the nonverbal information and learned by the model learning unit <NUM>. The nonverbal expression model <NUM> and the nonverbal recognition model <NUM> depend on the person, and thus different models are stored for respective users. In one example, model parameters of the nonverbal expression model <NUM> and the nonverbal recognition model <NUM> are adjusted to perform desired recognition of the nonverbal expression and conversion of nonverbal recognition, based on parameters of the extracted feature amount, for example. In another example, the model parameters of the nonverbal expression model <NUM> and the nonverbal recognition model <NUM> are adjusted according to the relation between the sender and the recipient based on the parameters of the extracted feature amount. On the other hand, the feature extraction model <NUM> and the conversion model <NUM> do not depend on the person, and thus one model is stored for each model. In another example, the storage unit <NUM> in which the learning models are stored is provided in one or more storage devices that are external to the information processing apparatus <NUM>.

Referring to <FIG>, the nonverbal information conversion system 1A is described according to the first embodiment. <FIG> is a schematic diagram illustrating an example of the nonverbal information conversion system 1A according to the first embodiment. The nonverbal information conversion system 1A according to the first embodiment is a system that converts nonverbal information using a nonverbal expression model of a sender and a nonverbal expression model of a person A in dialogue communication between the sender and a recipient. The person A is a person who is different from the sender and the recipient and who practices behavior that gives a better impression than the sender. The person A is an example of a third user.

First, the information processing apparatus <NUM> acquires video data in which the sender appears and stores the video data in the data storage unit <NUM> of the nonverbal information processing unit <NUM>. Next, the nonverbal information processing unit <NUM> reads the nonverbal expression model of the sender, and estimates the intention of the sender from the video data of the sender stored in the data storage unit <NUM> using the read nonverbal expression model of the sender. Then, the nonverbal information processing unit <NUM> stores information indicating the estimated intention of the sender in the intention storage unit <NUM>.

Next, the nonverbal information processing unit <NUM> reads the nonverbal expression model of the person A, and converts the feature amount of the nonverbal information indicated in the video data of the sender from the intention of the sender stored in the intention storage unit <NUM> using the read nonverbal expression model of the person A. Further, the nonverbal information processing unit <NUM> converts the video data of the sender based on the converted feature amount, to generate conversion data, which is synthesis information in which the video and label information are combined. The nonverbal information processing unit <NUM> stores the synthesis information, which is the conversion data, in the synthesis information storage unit <NUM>. Then, the information processing apparatus <NUM> outputs, to the recipient, the converted nonverbal information indicated by the synthesis information stored in the synthesis information storage unit <NUM> of the nonverbal information processing unit <NUM>.

As described, in dialogue communication between the sender and the recipient, the nonverbal information conversion system 1A according to the first embodiment converts nonverbal information using the nonverbal expression model of the person A, who expresses nonverbal information in easy-to-understand manner, in addition to the nonverbal expression model of the sender. This configuration improves the conversion accuracy of the nonverbal information.

Referring to <FIG>, processes or operation performed by the nonverbal information conversion system 1A is described according to the first embodiment. First, referring to <FIG>, an overall flow of a nonverbal information conversion process performed by the nonverbal information conversion system 1A is described. <FIG> is a flowchart illustrating an example of the nonverbal information conversion process.

First, the data acquisition unit <NUM> of the information processing apparatus <NUM> acquires video data in which the sender appears, who is an example of a first user (step S1). Specifically, the transmission apparatus <NUM> used by the sender captures the sender using the camera <NUM>, and transmits the captured video data to the information processing apparatus <NUM>. Then, the data acquisition unit <NUM> of the information processing apparatus <NUM> acquires the video data transmitted from the transmission apparatus <NUM>. The video of the sender includes mainly, for example, video obtained by capturing actions performed by the sender in daily communication. The data acquisition unit <NUM> stores the acquired video data in the data storage unit <NUM> of the nonverbal information processing unit <NUM>.

Next, the nonverbal information processing unit <NUM> of the information processing apparatus <NUM> inputs the video data acquired in step S1 to the feature extraction model <NUM>, to extract a feature amount of the nonverbal information (step S2). The feature amount of the nonverbal information includes a parameter such as positions of facial landmarks of a person, an action unit (AU), line-of-sight angles of both eyes, positions of skeleton landmarks, a head rotation angle, or a distance.

Next, the nonverbal information processing unit <NUM> of the information processing apparatus <NUM> inputs the feature amount extracted in step S2 to the nonverbal expression model <NUM> to estimate the intention of the sender. For example, the nonverbal information processing unit <NUM> acquires intention label information indicating the intention of the sender using the nonverbal expression model <NUM> into which the extracted feature amount is input. The intention label information includes information in which the intensities of the basic six emotions "happy, angry, sad, surprised, disgusted, and composed" and "Neutral, Deny, Accept, Arousal, and Interested" are represented by the value from <NUM> to <NUM>.

Next, the nonverbal information processing unit <NUM> of the information processing apparatus <NUM> inputs the intention estimated in step S3 to the conversion rule set by the conversion rule setting unit <NUM> to convert the feature amount, thereby acquiring desired feature amount (step S4).

Next, the nonverbal information processing unit <NUM> of the information processing apparatus <NUM> inputs the feature amount converted in step S4 to the conversion model <NUM>, to convert the video data acquired in step S1, thereby acquiring the converted video as the conversion data (step S5).

Then, the data output unit <NUM> of the information processing apparatus <NUM> outputs the converted video converted in step S5, which is the conversion data, to the recipient, who is an example of a second user (step S6). Specifically, the data output unit <NUM> transmits the converted video to the reception apparatus <NUM> used by the recipient. The reception apparatus <NUM> displays the converted video transmitted (output) from the information processing apparatus <NUM> on the display <NUM>.

A detailed description is now given of each of the processes illustrated in <FIG> with reference to <FIG>. First, referring to <FIG>, the process of extracting a feature amount performed by the nonverbal information processing unit <NUM> of the information processing apparatus <NUM> in step S2 is described. <FIG> is a flowchart illustrating an example of a process of extracting a feature amount of nonverbal information.

First, the feature extraction unit <NUM> inputs a general person video, which is video obtained by capturing a person who is different from the sender and the person A (step S21). The general person video is video obtained by capturing a third party person (general person), who is different from the sender, the person A, and the recipient. Further, the general person video is video obtained by capturing a motion in which the general person variously changes a facial expression, a body or head direction, a line-of-sight direction, and a distance from the camera, for example. Next, the feature extraction unit <NUM> performs an annotation processing using the general person video input in step S21 (step S22). The feature extraction unit <NUM> defines a feature amount of the nonverbal information as label information by, for example, an input operation to the keyboard <NUM> by an annotator. Thus, the feature extraction unit <NUM> creates a data set used for the feature amount extraction process. The feature extraction unit <NUM> stores the created data set in the data storage unit <NUM>.

Next, the feature extraction unit <NUM> builds the feature extraction model <NUM> used for feature amount extraction (step S23). The feature extraction model <NUM> includes layers of an input layer, a convolutional neural network (CNN) layer, a long short-term memory (LSTM) layer, and an estimation layer. In the feature extraction model <NUM>, a feature amount of an image of each of frames of the input video is extracted by the CNN layer. Further, in the feature extraction model <NUM>, a feature amount and state information of the nonverbal information in each of the frames are extracted by the LSTM layer with the feature amount of a plurality of frame images in the input video as an input. The state information in each of the frames corresponds to memory information to be transmitted to the next cell in processing in the LSTM, and represents context information in the LSTM that performs text analysis. The state information represents a state of a person such as being still or in motion. By combining the CNN layer and the LSTM layer, the feature extraction model <NUM> concurrently performs an analysis of each frame image and an analysis of time-series information.

Next, the feature extraction unit <NUM> reads the data set created by the processes of step S21 and step S22 to learn the feature extraction model <NUM> (step S24). Thus, the model learning unit <NUM> learns the feature extraction model <NUM> used for extracting the feature amount. Since the extraction of the feature amount of the nonverbal information from the input video does not change depending on the person, it is enough that the learning of the feature extraction model <NUM> is performed once. In another example, in a case that the feature extraction model <NUM> that has already been learned is present, the processes of step S23 and step S24 are omitted.

Next, the feature extraction unit <NUM> inputs the video data acquired in step S1 to the feature extraction model <NUM> learned in the processes of step S23 and step S24 (step S25). Then, the feature extraction unit <NUM> acquires a feature amount of the nonverbal information indicated by the video data acquired in step S1 (step S26).

In another example, the above-described processes of step S21 to step S25 are omitted by using a known technology capable of acquiring a feature amount in real time, such as OpenFace described in<NPL>.

Next, referring to <FIG> and <FIG>, the process of estimating an intention performed by the nonverbal information processing unit <NUM> of the information processing apparatus <NUM> in step S3 is described. <FIG> is a flowchart illustrating an example of a process of estimating an intention of the sender.

First, the intention estimation unit <NUM> inputs the video data of the sender acquired in step S1 (step S31). Then, the intention estimation unit <NUM> performs an annotation processing using the video data input in step S21 (step S32). The intention estimation unit <NUM> defines an intention corresponding to the video of the sender as the intention label information by, for example, an input operation to the keyboard <NUM> by an annotator. Equation <NUM> is an example of the intention label information L.

Further, the annotate in step S32 is assumed to be performed by the sender himself/herself (annotator=sender) and defines the type and intensity of the intention as indicated in Equation <NUM>, for example. The annotator reproduces, for example, the video data input in step S31 and inputs a numerical value of the intensity of the intention in each frame of the video. In the annotation processing, for example, a dedicated application is used to allow the annotator to specify the numerical value of the intensity of the intention in each frame by a drag operation using an input device such as the pointing device <NUM>, thereby reducing workload of the annotation processing. In another example, the annotation is performed by specifying various types or multidimensional information such as the type or certainty factor of the intention in addition to specifying the numerical value of the strength of the intention by the drag operation or the like using the input device such as the pointing device <NUM>. Thus, the intention estimation unit <NUM> creates a data set used for the intention estimation process. The intention estimation unit <NUM> stores the created data set in the data storage unit <NUM>.

Next, the intention estimation unit <NUM> builds the nonverbal expression model <NUM> for the sender used for estimation of the intention (step S33). Since the expression of the intention from the feature amount of the nonverbal information depends on the person, the intention estimation unit <NUM> builds the nonverbal expression model <NUM> for the sender. The structure of the nonverbal expression model <NUM> is the same regardless of the person, and is implemented by layers of an input layer, an LSTM layer, and an estimation layer. As preprocessing, the intention estimation unit <NUM> inputs the feature amount of the nonverbal information extracted in step S26 to the nonverbal expression model <NUM>. Using the feature amounts of the plurality of frame images in the input video as an input, the intention estimation unit <NUM> outputs an intention and a frame number in each frame by the LSTM layer of the nonverbal expression model. The frame number indicates the ordinal number of an input frame among a plurality of frames indicating intentions.

The nonverbal expression model <NUM> is a learning model indicating a relation between an intention and a feature amount of nonverbal information, and has a structure such as a database, for example. In the disclosure, the database is referred to as an "intention-feature database" for the sake of explanatory convenience. <FIG> is a conceptual diagram illustrating an example of the intention-feature database corresponding to the nonverbal expression model. As illustrated in <FIG>, the feature amounts of the nonverbal information have time-series values for each intensity of the intention (<NUM>, <NUM>, <NUM>, in this example). Further, for the feature amounts ((<NUM>) to (<NUM>) illustrated in <FIG>) of the nonverbal information, there are stochastically four possible time-series values for each intensity of the intention (occurrence probability: <NUM>, <NUM>, <NUM>, <NUM>).

For example, the parameter (<NUM>) corresponding to AU1 (ActionUnit1 is expressed by the following Equation <NUM>. Parameter (<NUM>) includes a value for N frames (for example, N=<NUM>) for each of the four occurrence probabilities, as indicated in Equation <NUM>. The other parameters (<NUM>) to (<NUM>) are also expressed as in Equation <NUM> in substantially the same manner as the parameter (<NUM>). Accordingly, for example, when only one frame having the occurrence probability of <NUM> is extracted for the parameters (<NUM>) to (<NUM>), the following Equation <NUM> is obtained. <MAT> <MAT>.

Next, the intention estimation unit <NUM> reads the data set created by the processes of step S31 and step S32 to learn the nonverbal expression model <NUM> (step S34). Thus, the model learning unit <NUM> learns the nonverbal expression model <NUM> used for estimating the intention. Since expression of an intention from a feature amount of nonverbal information depends on the person, learning of the nonverbal expression model <NUM> is performed for each person.

Next, since the expression of an intention from the feature amount of nonverbal information depends on the person, the intention estimation unit <NUM> reads the nonverbal expression model <NUM> for the sender learned in step S33 and step S34 (step S35). Then, the intention estimation unit <NUM> estimates the intention of the sender based on the nonverbal information indicated by the video data acquired in step S <NUM> (step S36). The estimated intention includes multidimensional information such as the type, strength, or certainty factor of the intention. The intention estimation unit <NUM> acquires the intention label information, which is an estimated value of the intention, obtained by, inputting video data of the sender to the nonverbal expression model <NUM> for the sender, for example. The intention estimation unit <NUM> stores information on the estimated intention in the intention storage unit <NUM>.

Thus, the intention estimation unit <NUM> estimates the intention of the sender based on the video data of the sender and the nonverbal expression model for the sender, to estimate the intention in consideration of differences in expression of the nonverbal information between individuals or between cultures.

Next, referring to <FIG>, <FIG>, the process of converting the feature amount performed by the nonverbal information processing unit <NUM> of the information processing apparatus <NUM> in step S4 is described. <FIG> is a flowchart illustrating an example of the feature amount conversion processes.

First, the conversion rule setting unit <NUM> sets a conversion rule for conversion of the nonverbal information (step S41). The conversion rule set by the conversion rule setting unit <NUM> includes items of a conversion item, a conversion ratio, a conversion destination, and the intention-feature database (see <FIG>) corresponding to the nonverbal expression model <NUM> of the conversion destination.

The conversion item indicates a type of a feature amount of the nonverbal information to be converted. Examples of the type of the feature amount of the nonverbal information include, but are not limited to, posture information, line-of-sight information, intensity of a facial expression for each emotion, and a head rotation angle. In the disclosure, the posture information refers to only a position corresponding to a backbone among the skeleton landmarks of the feature amount of the nonverbal information. Since the conversion item does not include all of the facial landmarks and the skeleton landmarks, individual differences in face and physique are maintained while back muscle and a line of sight are converted.

The conversion ratio is a ratio indicating how much the conversion item is to be converted to resemble the conversion destination. The conversion ratio is defined by a numerical value from <NUM> to <NUM>. For example, "<NUM>" is selected to make the conversion item is to resemble the person A, and "<NUM>" is selected to maintain the state before conversion.

Further, the conversion destination indicates a person who is the conversion destination. In the embodiment, for example, the person A, who practices behavior that gives a better impression than the sender is set as the conversion destination. Then, the relation between the intention of the person A and the feature amount of the nonverbal information is defined as the intention-feature database corresponding to the nonverbal expression model <NUM>. The relation is defined by building and learning the nonverbal expression model <NUM> of the person A in substantially the same manner as the processes illustrated in <FIG>. For example, the relation is defined by inputting video of the person A and performing the annotation processing by the person A of annotating an intention corresponding to each frame of the input video, in substantially the same manner as the creation of the data set performed in step S31 and step S32. The intention-feature database corresponding to the nonverbal expression model <NUM> of the person A has the same structure as the example illustrated in <FIG>. However, the parameters are different from those of the nonverbal expression model of the sender.

Then, the feature conversion unit <NUM> applies the conversion rule set in step S41 to convert the feature amount by processes from step S42 to step S44. Specifically, the feature conversion unit <NUM> selects the sender's intention estimated by the process of step S3 (step S42). The feature conversion unit <NUM> selects the type of intention, the intensity, and the frame number from the video data of the sender according to the estimated intention. For example, the feature conversion unit <NUM> selects the intention whose intensity is largest as the intention of the frame. In a case that there are a plurality of intentions having the largest value as in Equation <NUM> described above, both of the intentions having the largest value are selected. Then, linear interpolation described below is performed to calculate a feature amount of nonverbal information between the selected intentions. In this case, the ratio is <NUM> for both intentions. The feature conversion unit <NUM> selects rows (records) of the corresponding intention from the nonverbal expression model illustrated in <FIG>, and selects a feature amount of nonverbal information of the corresponding frame number from the corresponding rows (records).

Next, the feature conversion unit <NUM> stochastically selects (stochastic selection) one of four possible time-series values as indicated in Equation <NUM> described above (step S43). Then, the feature conversion unit <NUM> performs linear interpolation of the feature amounts of the nonverbal information of the sender and the conversion destination using the following Equation <NUM> according to the conversion ratio indicated in the conversion rule set in step S41 (step S44). In Equation <NUM>, X1 represents the feature amount of the nonverbal information of the sender, X2 represents the feature amount of the nonverbal information of the conversion destination, and α represents the conversion ratio.

<FIG> a table indicating an example of parameters of the feature amounts of nonverbal information to be input. <FIG> indicates an example of a part of parameters of the feature amounts of the nonverbal information extracted from video data of the sender before conversion and a frame number estimated therefrom. For example, the feature conversion unit <NUM> converts the feature amount based on the intention-feature database illustrated in <FIG> corresponding to the nonverbal expression model using the parameters indicated in <FIG> as an input.

For example, it is assumed that the feature conversion unit <NUM> selects {Neutral, <NUM>} as the intention of the sender in step S42, and stochastically selects the time-series value {}<NUM> in step S43. Further, it is assumed that the conversion items indicated in the set conversion rule are only (<NUM>) to (<NUM>) illustrated in <FIG>, and the conversion ratio is α=<NUM>. <FIG> is a table indicating an example of parameters of the feature amounts of the nonverbal information converted in this case. As indicated in <FIG>, the parameters (<NUM>) to (<NUM>) are converted from the parameters of <FIG>, while the parameters (<NUM>) to (<NUM>) remain unchanged from the parameters before conversion. In other words, the feature conversion unit <NUM> converts only the values of the posture and the line-of-sight direction without changing the facial expression of the sender.

Thus, the feature conversion unit <NUM> converts the parameters of the feature amount of the nonverbal information based on the estimated value of the intention estimated by the intention estimation unit <NUM> and the conversion rule set by the conversion rule setting unit <NUM> so as to increase the probability that the intention of the sender is correctly communicated to the recipient.

Next, referring to <FIG>, the process of converting the video data performed by the nonverbal information processing unit <NUM> of the information processing apparatus <NUM> in step S5 is described. <FIG> is a flowchart illustrating an example of the video data conversion processes.

First, the video conversion unit <NUM> inputs general person video and a feature amount of nonverbal information corresponding to the general person video (step S51). The general person video that is input in step S51 is mainly video obtained by capturing the general person's motion of changing only the feature amount of nonverbal information variously. Examples of the motion of changing only the feature amount include, but are not limited to, a motion of changing the direction of the line of sight and a motion of changing the direction of the head. In this case, the label information is not required. Thus, the video conversion unit <NUM> creates a data set for performing the conversion process of the video data. The video conversion unit <NUM> stores the created data set in the data storage unit <NUM>.

Next, the video conversion unit <NUM> builds the conversion model <NUM> for converting the video data (step S52). The conversion model <NUM> is obtained by extending the GANimation method described in<NPL>, not only to facial expression but also to posture and line-of-sight information of a person. GANimation is a technology that inputs, to an image generation network, an intensity group of action unit (AU) feature amounts, which are facial expression labels, in addition to input video, to convert the video to video having a desired facial expression label. The conversion model <NUM> is implemented by extending the GANimation method to a feature quantity of nonverbal information in addition to the AU feature amount.

The video conversion unit <NUM> inputs a set of "video before conversion, feature amount of nonverbal information before conversion, and feature amount of desired nonverbal information obtained by conversion" to the conversion model <NUM>, and outputs a set of "video after conversion, and feature amount of nonverbal information obtained by conversion". This configuration is different from a general-purpose generative adversarial network (GAN) in that desired label information obtained by conversion is also input. The loss function is calculated by a mean squared error between the feature amounts of video obtained by conversion and nonverbal information obtained by conversion and the feature amounts of desired video and nonverbal information.

Next, the video conversion unit <NUM> reads the data set created by the process of step S51 to learn the conversion model <NUM> (step S53). Thus, the model learning unit <NUM> learns the conversion model <NUM> used for converting the video data. Since the conversion of the video data does not change depending on the person, it is enough that the learning of the conversion model <NUM> is performed once. In another example, in a case that the conversion model <NUM> that has already been learned is present, the processes of step S52 and step S53 are omitted.

Next, based on the conversion model read in step S53, the video conversion unit <NUM> converts the video data acquired in step S1 (step S54). The video conversion unit <NUM> converts the video data for each frame. The video conversion unit <NUM> stores, in the synthesis information storage unit <NUM>, the conversion data, which is synthesis information obtained by synthesizing the video and the intention label information.

Thus, the video conversion unit <NUM> converts the video data of the sender so as to increase the probability that the estimated value of the intention estimated by the intention estimation unit <NUM> is correctly communicated to the recipient based on the nonverbal expression model <NUM>.

As described, the nonverbal information conversion system 1A according to an unclaimed embodiment estimates the intention of the sender based on the video data of the sender and the nonverbal expression model for the sender, and converts the video data of the sender based on the estimated intention of the sender and the conversion rule that is based on the nonverbal expression model of a person who expresses nonverbal information in easy-to-understand manner. Thus, the nonverbal information conversion system 1A according to the first embodiment improves the conversion accuracy of nonverbal information for intelligibly communicating the intention that the sender wants to communicate to the recipient in a dialogue between the sender and the recipient.

Referring to <FIG>, <FIG>, a nonverbal information conversion system 1B is described according to another unclaimed embodiment. The same configurations and the same functions as those in the previous unclaimed first embodiment are denoted by the same reference numerals, and redundant descriptions thereof are omitted below. <FIG> is a schematic diagram illustrating an example of the nonverbal information conversion system 1B according to the present unclaimed embodiment. The nonverbal information conversion system 1B according to the present unclaimed embodiment is different from the nonverbal information conversion system 1A according to the previous unclaimed embodiment in that conversion of nonverbal information is performed using the nonverbal expression model of the sender and a specific correction value used for conversion in dialogue communication between the sender and the recipient. The nonverbal information conversion system 1B according to the present unclaimed embodiment sets the conversion rule by directly designating an item to be corrected and a correction guidance value instead of selecting a specific person as a conversion destination when converting the feature amount in step S4.

A description is now given of the feature amount conversion process of step S4 according to the present unclaimed embodiment, the process being performed in a different way from the previous unclaimed embodiment. In the present unclaimed embodiment, the conversion rule set by the conversion rule setting unit <NUM> in step S41 includes a conversion item, a value of the conversion item, and a conversion ratio. The conversion item indicates a type of a nonverbal information feature amount to be changed. The correction guidance value is provided as a value of the conversion item. For example, the conversion item and the correction guidance value are posture information (<NUM>,<NUM>) and line-of-sight information (<NUM>,<NUM>).

The conversion ratio is a ratio indicating how much the conversion item is to be converted to be similar to the conversion destination. As the conversion ratio, a relatively small value is defined in order to smoothly perform video conversion. The conversion ratio is, for example, "<NUM>".

<FIG> indicate an example in which the conversion rule set in step S41 is applied in the present unclaimed embodiment. <FIG> is a table indicating an example of a parameter of a feature amount of nonverbal information to be input. <FIG> is a table indicating an example of a parameter of a feature amount of nonverbal information obtained by conversion. The feature conversion unit <NUM> converts the feature amounts based on the conversion rule set in step S41 using the parameters indicated in <FIG> as an input, for example.

As indicated in <FIG>, the parameters of (<NUM>) to (<NUM>) are closed to the correction guidance value indicated in the conversion rule from the parameter of <FIG>, while the parameters of (<NUM>) to (<NUM>) remain unchanged from the parameters before conversion. In other words, the feature conversion unit <NUM> performs conversion so that, for example, the posture and the line of sight are close to the correction guidance value of correction regardless of the intention.

As described, the nonverbal information conversion system 1B according to the present unclaimed embodiment converts the video data of the sender based on the conversion rule using the estimated intention of the sender and the correction guidance value of the conversion item. Thus, the nonverbal information conversion system 1B according to the present unclaimed embodiment converts nonverbal information by designating a specific numerical value of an item to be converted, thereby converting the nonverbal information without being affected by the intention of the nonverbal information.

Referring to <FIG>, the nonverbal information conversion system 1C is described according to another unclaimed embodiment. The same configurations and the same functions as those in the above-described embodiments are denoted by the same reference numerals, and redundant descriptions thereof are omitted below. <FIG> is a schematic diagram illustrating an example of the nonverbal information conversion system 1C according to the present unclaimed embodiment. The nonverbal information conversion system 1C according to the present unclaimed embodiment is different from the nonverbal information conversion system 1A according to the previous unclaimed embodiments in that conversion of nonverbal information is performed using the nonverbal expression model of the sender and a nonverbal expression model of a general person in dialogue communication between the sender and the recipient. The nonverbal information conversion system 1C according to the present unclaimed embodiment sets the conversion rule by designating a general person as a conversion destination when converting the feature amount in step S4. The general person refers to a person having an intention-feature database corresponding to an average nonverbal expression model.

A description is now given of the feature amount conversion process of step S4 according to the present unclaimed embodiment, the process being performed in a different way from the previous embodiments. In the present unclaimed embodiment, the conversion rule set by the conversion rule setting unit <NUM> in step S41 includes items of a conversion item, a conversion ratio, a conversion destination, and the intention-feature database (see <FIG>) corresponding to the nonverbal expression model <NUM> of the conversion destination. The conversion item indicates a type of a feature amount of the nonverbal information to be converted. Examples of the type of the feature amount of the nonverbal information include, but are not limited to, posture information, line-of-sight information, intensity of a facial expression for each emotion, and a head rotation angle.

The conversion ratio is a ratio indicating how much the conversion item is to be converted to resemble the conversion destination. The conversion ratio is defined by a numerical value from <NUM> to <NUM>. For example, "<NUM>" is selected to make the conversion item is to resemble the person as the conversion destination, and "<NUM>" is selected to maintain the state before conversion. In the embodiment, for example, "<NUM>" is defined as the conversion ratio so that the conversion item resembles the general person.

Further, the conversion destination indicates a person who is the conversion destination. In the embodiment, the general person is set as the conversion destination. The conversion rule setting unit <NUM> defines a relation between an intention of the general person and the feature amount of the nonverbal information as the intention-feature database corresponding to the nonverbal expression model <NUM>. The relation is defined by building and learning the nonverbal expression model <NUM> of the general person in substantially the same manner as the processes illustrated in <FIG>. For example, the relation is defined by inputting video of a desired person and performing the annotation processing by the person himself/herself of annotating an intention corresponding to each frame of the input video, in substantially the same manner as the creation of the data set performed in step S31 and step S32. The nonverbal information processing unit <NUM> performs this definition of the relation a plurality of times and obtains an average of the respective feature amounts, to create the average as the intention-feature database corresponding to the nonverbal expression model <NUM> of the general person. The feature conversion unit <NUM> applies the conversion rule thus set by the conversion rule setting unit <NUM> to convert the feature amount. The processes after step S41 are performed in the same or substantially the same manner as described above referring to step S42 to step S44 in the first embodiment.

As described, even in a case that the nonverbal information conversion system 1C according to the present unclaimed embodiment converts video data of the sender is based on the estimated intention of the sender and the conversion rule that is based on the nonverbal expression model of the general person, the nonverbal information conversion system communicates the intention that the sender wants to communicate to the recipient in an easy-to-understand manner.

Referring to <FIG> and <FIG>, a nonverbal information conversion system 1D is described according to the invention.

The same configurations and the same functions as those in the above-described embodiments are denoted by the same reference numerals, and redundant descriptions thereof are omitted below. <FIG> is a schematic diagram illustrating an example of the nonverbal information conversion system 1D.

The nonverbal information conversion system 1D is different from the nonverbal information conversion system 1A according to a previous unclaimed embodiment in that conversion of nonverbal information is performed using the nonverbal expression model of the sender and a nonverbal recognition model of the recipient in dialogue communication between the sender and the recipient. The nonverbal information conversion system 1D according to the invention converts a feature amount of nonverbal information of the sender by using the nonverbal recognition model of the recipient when performing the conversion process of the feature amount in step S4 so as to make it easy for the recipient to recognize the intention.

A detailed description is now given of the feature amount conversion process of step S4 the process being performed in a different way from the previous embodiments. <FIG> is a flowchart illustrating an example of the feature amount conversion processes.

First, the feature conversion unit <NUM> inputs a general person video, which is video obtained by capturing a person who is different the sender and the recipient (step S101). The general person video is video obtained by capturing a third party person (general person), who is different from the sender and the recipient. The general person video includes mainly video obtained by capturing actions performed by the general person in daily communication. Next, the feature conversion unit <NUM> performs an annotation processing using the general person video input in step S101 (step S102). The feature conversion unit <NUM> defines an intention corresponding to the general person video as the intention label information by, for example, an input operation to the keyboard <NUM> by an annotator. This intention label information is the same or substantially the same as the example indicated in Equation <NUM> described above.

Further, the annotate in step S102 is assumed to be performed by the recipient himself/herself (annotator=recipient) and defines the type and intensity of the intention as indicated in Equation <NUM>, for example. The annotator reproduces, for example, the video data input in step S101 and inputs a numerical value of the intensity of the intention in each frame of the video. In the annotation processing, for example, a dedicated application is used to allow the annotator to specify the numerical value of the intensity of the intention in each frame by a drag operation using an input device such as the pointing device <NUM>, thereby reducing workload of the annotation processing. In another example, the annotation is performed by specifying various types or multidimensional information such as the type or certainty factor of the intention in addition to specifying the numerical value of the strength of the intention by the drag operation or the like using the input device such as the pointing device <NUM>. Thus, the feature conversion unit <NUM> creates a data set used for the feature amount extraction process. The feature conversion unit <NUM> stores the created data set in the data storage unit <NUM>.

Next, the feature conversion unit <NUM> builds the nonverbal recognition model <NUM> for the recipient used for feature amount conversion (step S103). Since the recognition of the intention from the feature amount of the nonverbal information depends on the person, the feature conversion unit <NUM> builds the nonverbal recognition model <NUM> for the recipient. The structure of the nonverbal recognition model <NUM> is the same regardless of the person, and is implemented by layers of an input layer, an LSTM layer, and an estimation layer. As preprocessing, the feature conversion unit <NUM> inputs the feature amount of the nonverbal information extracted in step S26 to the nonverbal recognition model <NUM>. Using the feature amounts of the plurality of frame images in the input video as an input, the feature conversion unit <NUM> outputs an intention and a frame number in each frame by the LSTM layer of the nonverbal recognition model <NUM>. The frame number indicates the ordinal number of an input frame among a plurality of frames indicating intentions.

The nonverbal recognition model <NUM> is a learning model indicating a relation between nonverbal recognition and expression, and has a structure such as a database, for example. In the embodiment, the database is referred to as an "intention-feature database", for the sake of explanatory convenience. The intention-feature database corresponding to the nonverbal recognition model <NUM> has the same structure as the structure of the intention-feature database of the nonverbal expression model <NUM> illustrated in <FIG>.

Next, the feature conversion unit <NUM> reads the data set created by the process of step S101 and step S102 to learn the nonverbal recognition model <NUM> (step S104). Thus, the model learning unit <NUM> learns the nonverbal recognition model <NUM> used for converting the feature amount. Since recognition of an intention from a feature amount of nonverbal information depends on the person, learning of the nonverbal recognition model <NUM> is performed for each person. Next, since the recognition of an intention from the feature amount of nonverbal information depends on the person, the feature conversion unit <NUM> reads the nonverbal recognition model <NUM> for the recipient learned in step S103 and step S104 (step S105).

Next, the conversion rule setting unit <NUM> sets a conversion rule for conversion of the nonverbal information (step S106). The conversion rule set by the conversion rule setting unit <NUM> includes items of a conversion item, a conversion ratio, and the intention-feature database (see <FIG>) corresponding to the nonverbal recognition model <NUM> of the conversion destination.

The conversion item and the conversion ratio indicate a type of a feature amount of the nonverbal information to be converted. Examples of the type of the feature amount of the nonverbal information include, but are not limited to, posture information, line-of-sight information, intensity of a facial expression for each emotion, and a head rotation angle. The conversion ratio is a ratio indicating how much the conversion item is to be converted to resemble the conversion destination. The conversion ratio is defined by a numerical value from <NUM> to <NUM>. For example, the value <NUM> is selected to make the conversion item is to resemble the recipient, and the value <NUM> is selected to maintain the state before conversion. Further, the relation between an intention that is easy for the recipient to recognize and the feature amount of the nonverbal information is defined as the intention-feature database corresponding to the nonverbal recognition model <NUM> build by the processes from step S101 to step S105.

Then, the feature conversion unit <NUM> applies the conversion rule set in step S106 to convert the feature amount by processes from step S107 to step S109. Processes from step S107 to step S109 are performed in the same or substantially the same manner as described above referring to step S42 to step S44 of <FIG>, and the redundant descriptions are omitted.

As described above, the nonverbal information conversion system 1D according to the fourth embodiment converts the video data of the sender based on the estimated sender's intention and the conversion rule that is based on the nonverbal recognition model of the recipient. Thus, the nonverbal information conversion system 1D uses both an intention that the sender wants to communicate and the recipient's recognition of nonverbal information, thereby improving the conversion accuracy of nonverbal information for intelligibly communicating the intention that the sender wants to communicate to the recipient.

As described, the nonverbal information conversion system <NUM> (1A, 1B, 1C, 1D) uses the nonverbal expression model and the nonverbal recognition model, each of which differs depending on the person, to convert nonverbal information represented by video data of the sender. With this configuration, the nonverbal information conversion system <NUM> converts the nonverbal information in consideration of individuality in expression and recognition of nonverbal information. Thus, the nonverbal information conversion system <NUM> (1A, 1B, 1C, 1D) converts nonverbal information in consideration of individuality of each person in dialogue communication, thereby intelligibly communicating an intention that the sender wants to communicate to the recipient.

The functionality of the elements disclosed herein may be implemented using circuitry or processing circuitry which includes general purpose processors, special purpose processors, integrated circuits, application specific integrated circuits (ASICs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), system on a chips (SOCs), graphics processing units (GPUs), conventional circuitry and/or combinations thereof which are configured or programmed to perform the disclosed functionality. Processors are considered processing circuitry or circuitry as they include transistors and other circuitry therein. In the disclosure, the circuitry, units, or means are hardware that carry out or are programmed to perform the recited functionality. The hardware may be any hardware disclosed herein or otherwise known which is programmed or configured to carry out the recited functionality. When the hardware is a processor which may be considered a type of circuitry, the circuitry, means, or units are a combination of hardware and software, the software being used to configure the hardware and/or processor.

Although the information processing apparatus, the nonverbal information conversion system, the information processing method, and the program according to embodiments of the present invention have been described above, the above-described embodiments are illustrative and do not limit the present invention. Thus, numerous additional modifications and variations are possible in light of the above teachings.

The data acquisition unit <NUM> is an example of acquisition means. The data output unit <NUM> is an example of output means. The feature extraction unit <NUM> is an example of feature amount extraction means. The intention estimation unit <NUM> is an example of intention estimation means. The feature conversion unit <NUM> is an example of nonverbal information conversion means. The video conversion unit <NUM> is an example of nonverbal information conversion means.

Claim 1:
An information processing apparatus (<NUM>) that converts nonverbal information, the information processing apparatus comprising:
intention estimation means (<NUM>) for estimating an intention indicated in first nonverbal information, which is nonverbal information of a first user, based on a feature of the first nonverbal information and a nonverbal expression model indicating a relation between the first nonverbal information and an intention; and
nonverbal information conversion means (<NUM>, <NUM>) for converting the first nonverbal information into second nonverbal information to be output to a second user based on the estimated intention and a set conversion rule of the nonverbal information,
characterized in that
the conversion rule is set based on a nonverbal expression model for the second user.