Patent ID: 12198701

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, one or more embodiments of a conversation support system, a conversation support method and a computer readable storage medium storing a program according to the present disclosure will be described with reference toFIG.1toFIG.5.

Although a variety of limitations that are technically favorable for carrying out the present invention are put on the following embodiment(s), the technical scope of the present invention is not limited to the following embodiment(s) or illustrated examples.

Overall Configuration of Conversation Support System

First, an outline of configuration of a conversation support system100according to an embodiment(s) will be described.FIG.1is a block diagram schematically showing overall configuration of the conversation support system100.

The conversation support system100is provided at an utterance place where utterance is delivered to a plurality of persons.

The conversation support system100of this embodiment is expected to be provided, for example, as shown inFIG.1, at an utterance place in which two participants A, B in a conference room R1, three participants C, D, E in a conference room R2and one participant F at home or in an open space of his/her destination or the like participate.

The conversation support system100of this embodiment is expected to be provided at a wide range of places/occasions where persons exchange their opinions or the like, such as a conference, a meeting, and a place of brainstorming.

Although not all of the participants A to F may utter (speak), they are each an “utterer” having a possibility of uttering. In this embodiment, there is a plurality of utterers each having a possibility of uttering.

Regardless of whether they utter, the participants A to F are each a “performer” having a possibility of marking (performing marking motions) to give markers to his/her own utterances (talks) or the other utterers' utterances.

In an example shown inFIG.1, in the conference room R1, one terminal device1(1a) and two microphones2as voice receivers are provided.

The microphones2provided in the conference room R1are provided for the respective participants A, B. That is, a microphone2ais provided for the participant A, and a microphone2bis provided for the participant B.

In the conference room R2, one terminal device1(1b) and one microphone2(2c) are provided. That is, in the conference room R2, one microphone2cis provided for the participants C, D, E.

The terminal devices1(1a,1b) provided in the conference rooms R1, R2are stationary/desktop personal computers (hereinafter referred to as “PC(s)”), laptop PCs, tablet PCs or the like.

The participant F has a portable terminal device (terminal device1c), such as a smartphone, and participates in the utterance place through the terminal device1c.

The terminal device1cincludes a microphone2(2d) as a voice receiver, a camera3as an imager described below, and a loudspeaker4as a voice outputter.

The terminal devices1a,1b, which are stationary/desktop PCs, laptop PCs, tablet PCs or the like, may each be provided with a microphone2, a camera3, a loudspeaker4and/or the like that may be built in each terminal device1or may be external components attached to each terminal device1.

In this embodiment, tile terminal devices1(1a,1b,1c) are connected to a server5via a communication network6.

The communication network6may be Internet or another network, such as a LAN (Local Area Network). The terminal devices1(1a,1b,1c) may be connected to the server5with cables.

Configuration of Terminal Device

Next, configuration of the terminal device(s)1of the conversation support system100according to this embodiment will be described with reference toFIG.2.

FIG.2is a block diagram showing configuration of the main part of the terminal device1(1a/1b).

The terminal device1cis, as described above, a portable terminal device, such as a smartphone, but not limited thereto, and hence may be a laptop PC, a tablet PC or the like having built-in microphone2, camera3, loudspeaker4and/or the like.

The terminal device1cis different from the terminal device in having built-in microphone2, camera3and loudspeaker4shown inFIG.2. Except this, the terminal device1cis the same as the terminal device1a/1bin configuration. Hence, hereinafter, the terminal device1a/1bis described, and description of the terminal device1cis omitted.

As shown inFIG.2, the terminal device1a/1bincludes a controller11, a storage12, an operation unit13, a display14, a communication unit15and a bus16that connects these components11to15with one another. To the terminal device1a/1b, a microphone(s)2as a voice receiver(s), a camera(s)3as an imager(s), a loudspeaker(s)4as a voice outputter(s) and or the like are connected.

The controller11includes a CPU (Central Processing Unit) and a RAM (Random Access Memory). In response to operations on/off the operation unit13, the CPU of the controller11reads various programs stored in the storage12, loads the read programs into the RAM, and performs various processes in accordance with the loaded programs, thereby performing concentrated control of operation of each component of the terminal device1.

The storage12is configured by a nonvolatile semiconductor memory, a hard disk and/or the like, and has a program storage region (not shown) where, for example, various programs to be executed by the controller11and parameters necessary to perform processes of the programs are stored, and a data storage region (not shown) where, for example, various data are stored.

In the data storage region of the storage12, the following data are stored: voice data of utterances received by the microphone2as a voice receiver, text data into which the voice data have been converted: image data of images of gestures performed by a performer(s) and caught by the camera3as an imager; and other data. In this embodiment, voice data of each utterance is stored in the storage12in association with identification information identifying the utterer of the utterance, and image data of each gesture is stored in the storage12in association with identification information identifying the performer of the gesture.

The operation unit13includes: a keyboard including cursor keys, number input keys and various function keys; and a pointing device, such as a mouse, and outputs, to the controller11instruction signals input by operations on/with the keys of the keyboard or the mouse. The operation unit13may include an operation button, such as a push button. Additionally or alternatively, an operation device (not shown) configured separately from the terminal device1and including an operation button or the like may be provided in the conversation support system100. In this case, this operation device also functions as the operation unit13.

The operation unit13may have a touchscreen on the display screen of the display14. In this case, the operation unit13outputs, to the controller11, instruction signals input via the touchscreen.

In this embodiment, voice data are converted into texts (text data), and the texts are displayed on the display14. When the touchscreen is laid on the display screen of the display14, marking motions may each be performed by selecting a character string (text) corresponding to an utterance from a list displayed on the display screen. The character string may be selected by a key operation or an operation button operation, for example. In this case, the operation unit13functions as a marking motion catcher with which a performers) inputs/performs marking motions and which catches/obtains information thereof.

The display14is configured by a monitor, such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube), and displays instructions input from the operation unit13, data and/or the like in accordance with instructions of display signals input from the controller11.

In this embodiment, the display14can display data in which voice data of utterances of participants and marking motions of the participants are associated (linked) with one another on the same timeline.

That is, in this embodiment, as described below, voice data are converted into text data of character strings, and the character strings are displayed in the form of a list on the display14, and these text data of the voice data and markers are associated with one another and displayed as the list as shown inFIG.5.

On the display screen of the display14, as described above, the touchscreen, which functions as the operation unit13, may be laid.

The communication unit15includes a LAN adapter, a modem and/or a TA (Terminal Adapter) to control data sending/receiving to/from the devices or the like that are connected to the communication network6.

The microphone2connected to the terminal device1of this embodiment records a voice when an utterer utters.

In this embodiment, when a performer performs a marking motion by voice, the microphone2records this voice too, and thereby functions as a marking motion catcher.

The microphone2constitutes a voice receiver together with an amplifier (not shown), an A/D converter (not shown) and/or the like, and converts voice data input as analog signals into digital data as needed. The voice data recorded by the microphone2and converted into the digital data are sent to the controller11.

In this embodiment, voice data are sent from the terminal device1to the server5, and obtained by a controller51described below.

The microphone2may be fixedly provided (installed) in the conference room R1/R2or may be portable. The microphone2may be used by being attached to the terminal device1. When there are two or more utterers (participants) each having a possibility of uttering, it is preferable that two or more microphones2be provided in the conversation support system1, and it is further preferable that microphones2be provided for the respective utterers (participants).

The camera3is an imager that images gestures of persons.

When a participant performs a marking motion, which is described below, by gesture, the camera3catches the gesture, and thereby functions as a marking motion catcher.

The camera3may be fixedly provided (installed) in the conference room R1/R2, or may be portable. The camera3may be used by being attached to the terminal device1.

When there are two or more performers each having a possibility of marking, it is preferable that two or more cameras3be provided in the conversation support system100, and it is further preferable that cameras3be provided for the respective performers.

When cameras3are provided for the respective performers, image data of images taken by the cameras3are associated with information (identification information, such as names or identification numbers) of the respective performers for whom the respective cameras3are provided. More specifically, preset identification information is attached to image data of each taken image. This makes it possible to obtain marking motions in association with their respective performers. That is, it is possible to identify, from image data of each taken image, the performer of each marking motion.

Data (image data of taken images) obtained by the camera3of the terminal device1are sent to the controller11thereof. In this embodiment, image data are sent from the terminal device1to the server5, and obtained by the controller51described below.

The loudspeaker4reproduces and outputs, for example, voice data of utterances made by an utterer(s).

The loudspeaker4constitutes a voice outputter together with a D/A converter (not shown), an amplifier (not shown) ail nor the like.

Connecting the loudspeakers4to the terminal devices I that are provided in conference rooms or the like at locations away from one another enables participants in the conference rooms or the like to participate in the utterance place, listen to the content of each utterance made by any participant, and have conversations with one another.

Configuration of Server

Next, configuration of the server5of the conversation support system100according to this embodiment will be described with reference toFIG.3.

The server5processes various data sent from the terminal devices1(1a,1b,1c) and sends the processing results to the terminal devices1(1a,1b,1c). This enables the participants A to F participating in the utterance place through the terminal devices1(1a,1b,1c) to have the same information.

FIG.3is a block diagram showing configuration of the main part of the server5.

As shown inFIG.3, the server5includes a controller51(hardware processor), a storage52, an operation unit53, a display54, a communication unit55and a bus56that connects these components51to55with one another.

The controller51includes a CPU (Central Processing Unit) and a RAM (Random Access Memory). In response to operations on/with the operation unit53, the CPU of the controller51reads various programs stored in the storage52, loads the read programs into the RAM, and performs various processes in accordance with the loaded programs, thereby performing concentrated control of operation of each component of the server5.

In this embodiment, the controller51functions as a voice obtaining unit, an utterer identification unit, a voice timeline creation unit, a marking analysis unit, a marking timeline creation unit, a conversion-into-text unit, a data linkage unit and/or the like.

The controller51(voice obtaining unit) obtains voice data of utterances made by utterers (participants A to F) and received by the microphones2as voice receivers. In this embodiment, voice data of utterances received by the microphones2of the respective terminal devices1are obtained by the controllers11of the respective terminal devices1. The voice data obtained by the controllers11are sent from the respective terminal devices1to the server5via the communication units15,55, and obtained by the controller51of the server5.

Further, the controller51(utterer identification unit) identifies the utterer (one of participants A to F) of each obtained utterance (voice data). That is, as in the case shown inFIG.1, when there are two or more utterers, the controller51identifies who has made which utterance (talk). Association of each utterance with the utterer of the utterance is stored in the storage52or the like.

The method for the controller51to identify the utterer of each utterance is not particularly limited.

For example, as in the conference room R1shown inFIG.1, when the microphones2are provided for the respective utterers (participants A, B), the controller51identifies the utterer of each utterance by obtaining correspondence between the microphones2and the utterers (participants A, B) for whom the respective microphones2are provided. For example, in advance, individual identification (ID) numbers or the like are given to the respective microphones2, and the microphone2ahaving an ID number of 001 and the microphone2blaving an ID number of 002 are registered to be used by the participant A and the participant B, respectively. This makes it possible to, when an utterance is made and received by a microphone2, attach, to voice data of the utterance, information indicating whether the microphone2that has received the utterance is the microphone2ahaving an ID number of 001 or the microphone2bhaving an ID number of 002. Identifying the utterer of each utterance on the basis of the information attached to voice data of the utterance makes it possible to easily identify who has made the utterance.

Alternatively, for example, the controller51may identify the utterer of each utterance by comparing a voice waveform of the utterance received by a microphones2with each of voice waveforms unique to the respective utterers. For example, in advance, voice waveforms of the respective utterers (participants C, D, F) are associated with the identification information (e.g. ID numbers) of the respective utterers. This makes it possible to, when an utterance is made and received by a microphone2, compare the voice waveform of the utterance with each of the voice waveforms of the utterers (participants C, D, E). This configuration makes it possible to identify the utterer of each utterance even when, as in the conference room R2shown inFIG.1, one microphone2is provided for a plurality of utterers (participants C, D, F).

Further, the controller51(voice timeline creation unit) manages each obtained voice data on a voice timeline. That is, to voice data of each utterance, time data of time at which the utterance was made is attached. The controller51arranges voice data of utterances in chronological order on the basis of their respective time data.

Further, the controller51(marking analysis unit) analyzes each marking motion by which a marker is given to an utterance, when the marking motion is caught.

The “marker” is a mark or a sign that is given to an utterance when a person who has listened to or made the talk (utterance) (i) judges the talk important, (ii) pays special attention to the talk, or (iii) is interested in the talk, for example.

When a participant performs a predetermined marking motion for an utterance, the controller51stores the marking motion in the storage52together with time data of the performed time of the marking motion.

Markers each indicating a plurality of meanings (performer's intentions), such as “important”, “valuable”, “caution” and “question”, may be given by the same type of marking motion. Alternatively, markers indicating different meanings may be associated with different types of marking motions so that why each marker has been given can be understood. In this case, the controller51analyzes each marking motion to obtain its meaning, attaches information on the performer's intention to the marking motion, and stores these in the storage52or the like, for example.

In this embodiment, as marking motions, for example, voice input, gesture input, and input with an operation means are assumed.

When marking motions are performed by voice, the microphone(s)2is the marking motion catcher, when marking motions are performed by gesture, the camera(s)3is the marking motion catcher, and when marking motions are performed by operating an operation means, the operation unit(s)13or the like is the marking motion catcher.

Information caught by the microphone2, the camera3and the operation unit13is not only marking motions. Hence, the controller51analyzes each motion to determine whether the motion is a marking motion.

For example, in the case of voice input, utterances (voices) as marking motions, such as “nice” and “mark”, are registered in advance, and when a caught utterance (voice) corresponds to one of these, the controller51determines that the utterance is not a mere utterance but a marking motion.

As another example, in the case of gesture input, gestures as marking motions, such as a gesture of right hand thumbs-up and a gesture of left hand waving, are registered in advance, and when a caught gesture corresponds to one of these, the controller51determines that the gesture is not a mere motion but a marking motion.

In this embodiment, multiple types of marking motions are prepared for, on the voice timeline, the position of each utterance to Which a marker is given.

That is, in this embodiment, there are prepared at least a first marking motion by which a marker is given at the uttered timing of an utterance (i.e. by which a marker is given to an utterance made at the performed timing of the marking motion) and a second marking motion by which a marker is given to an utterance made at a predetermined timing before the performed timing of the marking motion.

For example, in the case of voice input, when a voice of “mark” is caught, the controller51determines that the caught voice is the first marking motion by which a marker is given at the uttered timing of an utterance, whereas when a voice of “three mark” is caught, the controller51determines that the caught voice is the second marking motion by which a marker is given to an utterance made three seconds earlier than the performed timing of the marking motion.

As another example, in the case of gesture input, when a gesture of one-time left hand waving is caught, the controller51determines that the caught gesture is the first marking motion by winch a marker is given at the uttered timing of an utterance, whereas when a gesture of one-time right hand waving is caught, the controller51determines that the caught gesture is the second marking motion by which a marker is given to an utterance one utterance before an utterance made at the performed timing of the marking motion.

As another example, in the case of input with an operation means (operation button), the position, on the voice timeline, of an utterance to which a marker is given may be selected on the basis of at least one of: the number of times the operation button has been operated; the length of time for which the operation button has been operated; and the strength of the operation applied to the operation button.

Thus, the second marking motion may be a marking motion by which a marker is given to an utterance (talk) in the past, the utterance being specified by time; for example, a marking motion by which a marker is given to an utterance made some seconds earlier than the performed timing of the marking motion. Alternatively, the second marking motion may be a marking motion by which a marker is given to an utterance (talk) in the past, the utterance being specified by going back to the past in units of utterances (talks); for example, a marking motion by which a marker is given to an utterance made some utterance(s) earlier than the performed timing of the marking motion.

Thus, the controller51analyzes the content of each marking motion to obtain its meaning.

Correspondence between marking motions and their respective meanings is preset and stored/registered in the storage52or the like, and the controller51analyzes each marking motion on the basis of the stored information. Rules of analysis of marking motions may be customized by each performer and registered.

Further, the controller51(marking timeline creation unit) manages each caught marking motion on a marking timeline.

As described above, when the first marking motion, by which a marker is given at the uttered timing of an utterance, and the second marking motion, by which a marker is given to an utterance made at a predetermined timing before the performed timing of the marking motion, are prepared, the controller51gives each marker to a position on the marking timeline, which follows the predetermined rules, on the basis of the above-described analysis result.

Further, the controller51(data linkage unit) links utterances with marking motions on the same timeline.

When, as described above, the controller51gives each marker to a position on the marking timeline, which follows the predetermined rules, the controller51reflects this created marking timeline in linking utterances with marking motions.

Further, in this embodiment, on the display14, text data of utterances (character strings of text data into which voice data of utterances have been converted) and markers given to tile utterances are displayed. When notes, comments or the like are written on the display14, the controller51links these with utterances and marking motions on the same timeline.

Further, when image data of images taken at the utterance place are present, the controller51also links these with utterances and marking motions on the same timeline.

Further, the controller51(conversion-into-text unit) converts each obtained voice data into text data.

More specifically, a dictionary database (not shown) or the like for converting utterances into texts (text data) is stored in the storage52or the like, and the controller51converts voice data of each utterance into text data, referring to the stored information. Voice data of fillers, such as “like”, “uh”, “well”, “umm” and “hmm”, may be excluded from the target of conversion into text data.

In this embodiment, the controller51punctuates and converts voice data of each utterance talk by talk into text data. More specifically, the controller51converts a range of voice data of an utterance determined as one talk (first talk) into text data (one text data), and when determining that the following forms another talk, separates it from the first talk. Thus, in the case where voice data of an utterance is divided into talks, text data thereof are displayed at different lines on the display14by line feed.

The method for the controller51to punctuate voice data of an utterance talk by talk is not particularly limited. For example, the controller51may punctuate voice data of an utterance talk by talk in response to detecting a silent state or detecting context.

That is, when determining that the silent state has continued for a predetermined time, or determining that the context has changed, the controller51determines, referring to the dictionary database or the like, that another talk starts, and separates it from the previous talk (first talk).

When the silent state is detected to determine whether an utterance is one talk, a silent-state continuing time (threshold), by which whether an utterance is one talk is determined, is preset. For example, when the silent state continues for two or more seconds, the controller51determines that another talk starts.

The storage52is configured by a nonvolatile semiconductor memory, a hard disk and/or the like, and has a program storage region (not shown) where, for example, various programs to be executed by the controller51and parameters necessary to perform processes of the programs are stored, and a data storage region (not shown) where, for example, various data are stored. In the program storage region of this embodiment, a program for conversation support is stored, for example.

In the data storage region of the storage52, the following data are stored: voice data of utterances received by the microphones2as voice receivers, text data into which the voice data have been converted; image data of images of gestures performed by performers and caught by the cameras3as imagers; and other data. In this embodiment, voice data of each utterance is stored in the storage52in association with identification information identifying the utterer of the utterance, and image data of each gesture is stored in the storage52in association with identification information identifying the performer of the gesture.

The operation unit53includes: a keyboard including cursor keys, number input keys and various function keys; and a pointing device, such as a mouse, and outputs, to the controller51, instruction signals input by operations on/with the keys of the keyboard or the mouse. The configuration of the operation unit53is not limited thereto. When various operation buttons are provided, and/or a touchscreen is laid on the display54, these also function as the operation unit53.

The display54is configured by a monitor, such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube), and displays instructions input from the operation unit53, data and/or the like in accordance with instructions of display signals input front the controller51.

The communication unit55includes a LAN adapter, a modem and/or a TA (Terminal Adapter) to control data sending/receiving to/from the devices or the like that are connected to the communication network6.

Operation of Conversation Support System

Next, a conversation support method realized by the conversation support system100according to this embodiment will be described with reference toFIG.4andFIG.5.

FIG.4is a flowchart of the conversation support method according to this embodiment.

As shown inFIG.4, when conversation support is performed by the conversation support system100, the controller51determines whether a microphone(s)2provided in the conversation support system100has been powered (i.e. is ON) (Step S1). When no microphone2has been powered (Step S1; NO), the controller51repeats Step1until a microphone(s)2is powered.

When a microphone(s)2has been powered (Step S1; YES), the controller51determines whether anyone (terminal devices1) has participated in the conversation support system100(Step S2). More specifically, the controller51determines whether person(s) (participant(s)) have expressed their participations in a conference, a meeting or the like by performing input operations representing the participations, such as logins. When no one has participated in the conversation support system100(Step S2, NO), the controller51repeats S2until person(s) participate in tile conversation support system100.

When person(s) have participated in the conversation support system100(Step S2; YES), the controller51starts obtaining voice data of utterances received by the microphones2provided in the conversation support system100(e.g. when the microphones2a,2b,2c,2dhave participated in the conversation support system100, alt of these microphones2) (Step S3).

When voice data of an utterance is obtained, the controller51identifies the utterer of the utterance (Step S4).

The controller51also determines whether the utterance is one talk (Step S5). If the controller51determines that the utterance is one talk (Step S5; YES), the controller51converts the voice data into text data as one talk (Step S6). If the controller51determines that the utterance is not one talk (Step S5; NO), the controller51punctuates and converts the voice data talk by talk into text data (Step S7).

The controller51manages voice data of each utterance on the voice timeline (i.e. in chronological order) by its uttered time on the basis of time data or the like attached to the voice data (Step S8).

Once person(s) have participated in the conversation support system100(Step S2, YES), the controller51determines at all times whether any marking motion has been caught for any utterance (Step S9) in parallel with the obtainment of voice data of utterances.

The controller51repeats Step S9until determining that a marking motion leas been caught, and when determining that a marking motion has been caught (Step S9, YES), identifies the performer of the marking motion (Step S10). When multiple types of marking motions having different meanings are prepared (registered in the storage52or the like), the controller51analyzes the caught marking motion to obtain its meaning.

When identification of the performer of the marking motion and analysis of the marking motion finish, the controller51proceeds to Step S11. The controller51manages each marking motion on the marking timeline (i.e. in chronological order) by its performed time (Step S11), When, as described above, the marking motion has been analyzed for an utterance made a predetermined time earlier than the performed time of the marking motion due to predetermined rules, the marking motion is arranged at a position on the marking timeline in accordance with the rules.

When utterances and marking motions are arranged on their timelines, the controller51links the utterances with the marking motions on the same timeline (Step S12).

Then, the coworker51causes the display14of each terminal device1participating in the conversation support system100to display the text data of the utterances and marker data based on the marking motions in chronological order (Step S13). More specifically, the controller51generates display data for displaying a list where the text data of the utterances and the marker data based on the marking motions are arranged in chronological order, and sends the generated display data to each terminal device1, and the controller11of each terminal device1controls display of the display14on the basis of the display data, and thereby causes the display14to display the list as shown inFIG.5on its display screen.

FIG.5shows an example of an output that is output during or after use of the conversation support system100of this embodiment. The output shown inFIG.5may be displayed, for example, on the display14of each terminal device1, or may be printed out.

InFIG.5, in an item of “INDEX”, sequential numbers of utterances (talks) are shown, in an item of “DATETIME”, uttered times of utterances (talks) are shown, in an item of “USERNAME”, login names (which may be real names or pseudonyms used in the conversation support system100) of participants are shown, in an item of “MARKER”, markers (marking log/history) given by participants, by performing marking motions, to talks that they think important or the like are shown, in an item of “LINK”, notes or the like linked with talks are shown, and in an item of “TALK”, texts of talks (utterances) are shown.

InFIG.5, in the items of “USERNAME” and “MARKER” of the participant A, emphasis is not placed (as markers, white stars are used), whereas in the items of “USERNAME”, “MARKER” and “LINK” of the participant B, boldface is used (as markers, black stars are used). This distinction makes it easy to visually recognize Who has put a mark or left a note on which talk. InFIG.5, for the sake of convenience of illustration, the distinction is made by boldface. When the output is actually displayed on the display14or printed out, it is preferable that different colors be used for different participants (i.e. color coding be performed), for example.

Further, inFIG.5, the distinction is made in the items of “USERNAME”, “MARKER” and “LINK” by the same method, namely by boldface (by color coding or the like), but the distinction may be made in other items too, such as the item of “TALK”, by the same method. However, since talks in the item of “TALK” and names of utterers in the item of “USERNAME” are displayed at the same points of time (on the same lines), the distinction in the item of “TALK” by color coding or the like is dispensable. Further, if it is important to know to winch talk many markers have been given, distinction of performers by color coding or the like may be unnecessary.

The method for distinguishing markers by performer of marking is not limited to the method of displaying markers in colors set for the respective performers. For example, following the respective markers, names (login names, etc.) of performers may be displayed. As another example, cartoon character marks or the like set for the respective performers may be displayed. For example, a cartoon drawing of a cat is displayed as markers given by marking motions performed by the participant A, and a cartoon drawing of a flower is displayed as markers given by marking motions performed by the participant B.

InFIG.5, only star-shaped markers are shown in the item of “MARKER”. However, as described above, when multiple types of markers indicating different meanings are prepared, markers of different shapes may be displayed as these different types of markers. For example, when a performer thinks of an utterance (talk) as an important point, a star-shaped marker may be displayed as a marker for “important”, whereas when a performer has a question about an utterance (talk), a triangular marker may be displayed as a marker for “question”.

In the item of “LINK”, for example, notes (texts) entered by participants into the terminal devices1are shown.FIG.5shows a note made by the participant B talk having an index of “249” and uttered by the participant A. When a participant enters a note or the like and thereafter selects a talk desired to be linked with, the note can be linked with the talk.

In the item of “LINK”, not only notes but also image data of taken images or the like may be linked with desired talks. In this case, for example, an icon indicating that image data is linked with a talk, a linked address (URL) and/or the like may be displayed in the item of “LINK”.

Further, data for output (display data or printing data) as shown inFIG.5, which is generated by the server5and sent to each terminal device1participating, in the conversation support system100, may be stored in the storage12or the like of each terminal device1. However, in terms of degree of freedom in use of each terminal device1and data management cost, it is preferable that the data for output be stored in the storage52or the like of the server5so as to be readable by each terminal device1as needed.

Thus, the conversation support system100of this embodiment enables participants to put important points or the like on record as needed by giving markers or the like thereto while concentrating on discussions and uttering, without requiring the participants to take either time or effort.

Advantageous Effects

As described above, the conversation support system100according to this embodiment is a conversation support system provided at an utterance place where utterance is delivered to a plurality of persons each being an utterer having a possibility of uttering and/or a performer having a possibility of marking, wherein the controller51obtains voice data of each utterance made by an utterer and received by a microphone2as a voice receiver, and manages the voice data on a voice timeline, and when a marking motion by which a marker is given to the utterance is caught, manages the marking motion on a marking timeline, and links the marking motion with the utterance on the same timeline.

This enables participants to, while concentrating on discussions, conversations or the like, give markers to utterances (talks) appropriately when they think the utterances (talks) important or the like. Regarding to which utterance (talk) each marker is given, the system side (in this embodiment, the controller51of the server5) automatically associates these with one another.

Thus, participants do not need to first record conversations and later create text data thereof and search for important talks from a list displayed. That is, participants can both concentrate on discussions and put important points on record without taking time or effort.

Further, in this embodiment, a plurality of utterers each having a possibility of uttering are present, and a plurality of microphones2as voice receivers are provided in the conversation support system100.

This enables smooth utterance even when a plurality of persons participate in the conversation support system100and utter simultaneously.

Further, the controller51identifies tile utterer of each utterance.

This makes it possible to identify who has made each utterance (talk) even when a plurality of persons participate in the conversation support system100.

In particular, when a plurality of microphones2as voice receivers are respectively provided for a plurality of utterers, the controller51can identify the utterer of each utterance by obtaining a correspondence between the microphones2and the utterers.

This makes it possible to automatically identify the utterer of each utterance without, before uttering, the utterer entering or declaring that he/she is about to utter, and hence enables smooth progress of conversations at a conference or the like.

Alternatively, the controller51may identify the utterer of each utterance by comparing the voice waveform of the utterance received by a microphone2with each of the voice waveforms unique to the respective utterers. This makes it possible to identify the utterer of each utterance (talk) accurately even when microphones2are not provided for the respective utterers.

Further, microphones2may each be portable. This enables participants to participate in the conversation support system100from anywhere, and improves degree of freedom in use of the conversation support system100.

Further, in this embodiment, the controller51converts voice data of each utterance into text data.

This makes it possible to easily create a book of minutes or the like without requiring participants to transcribe recorded utterances as text data.

Further, in this embodiment, the controller51punctuates and converts voice data of each utterance talk by talk into text data.

This makes it possible to generate text data that are easy to read, so that the content of a conference of conversations can be checked later.

Further, the controller51punctuates voice data of each utterance talk by talk in response to detecting the silent state or detecting the context.

This makes it possible to punctuate voice data of each utterance talk by talk at appropriate points.

Further, in this embodiment, a plurality of performers having a possibility of marking are present, and a plurality of marking motion catchers, such as cameras3, are provided in the conversation support system100.

This makes it possible to catch marking motions appropriately even when a plurality of persons judge the content of utterances.

In particular, when a plurality of marking motion catchers, such as cameras3, are respectively provided for a plurality of performers, and marking motions are obtained in association with their respective performers, it is easy to know who has paid special attention to which utterance (talk), for example.

Marking motions may each be performed by voice. In this case, a microphone(s)2as a voice receiver(s) serves as a marking motion catcher(s).

This enables participants to perform marking motions smoothly without being away from microphones2that are used for uttering.

Marking motions may each be performed by gesture, in this case, a camera(s)3as an imager(s) serves as a marking motion catcher(s).

This enables participants to perform marking motions easily even while uttering by performing gestures, without interrupting conversations or the like.

Marking motions may each be performed by selecting a character string corresponding to an utterance from a list of character strings of text data into which voice data of utterances have been converted, the list being displayed on the display14of each terminal device1. This enables participants to easily give, to utterances (talks), markers for which the participants have forgotten to perform marking motions, while checking the display screen of the display14.

Marking motions may each be an input operation (man operation button of each terminal device1provided in the conversation support system100, or an operation button of an operation device or the like provided separately from the terminal device1.

This enables participants to perform marking motions by operating a button(s).

Multiple types of marking motion catchers described above may be provided in the conversation support system100. This enables participants to select how to perform marking motions according to the scene, and hence perform marking motions more freely.

In this embodiment, when notes, comments or the like are written on the display screen or the like of the display14of each terminal device1, the controller51links the written notes or the like with utterances and marking motions on the same timeline.

This makes it possible to easily associate and put on record utterances with what participants come up with or are aware of in the middle of conversations or the like.

Further, when images are taken at the utterance place, the controller51links image data of the taken images with utterances and marking motions on the same timeline.

This makes it possible to easily associate and put on record utterances with image data of taken images even when characters or the like written as notes on a board or the like are imaged.

Further, in this embodiment, multiple types of marking motions may be prepared for, on the voice timeline, positions of utterances to which markers are given.

For example, the multiple types of marking motions may include a first marking motion by which a marker is given at the uttered timing of an utterance and a second marking motion by which a marker is given to an utterance made at a predetermined timing before the performed timing of the marking motion.

This enables participants to appropriately give markers to utterances to which they desire to give markers, even when they could not perform marking motions promptly at the uttered timings of the utterances.

For example, when marking motions are each performed by operating an operation button, the position, on the voice timeline, of an utterance to which a marker is given may be selected on the basis of at least one of the number of times the operation button has been operated, the length of time for which the operation button has been operated, and the strength of the operation applied to the operation button.

This makes it possible to relatively easily and accurately select an utterance (talk) to which a marker is given.

Further, in this embodiment, the controller51links identification information identifying the performer of each marking motion with the marking motion.

This makes it possible to know who has paid special attention to which utterance (talk), for example.

In particular, when text data of utterances and markers given to the utterances are displayed, and also the identification information identifying the performer of each marking motion by which a marker has been given is displayed, it is easy to know who has paid special attention to which utterance (talk), for example.

Further, when, as the identification information, names of performers Of colors set for the respective performers are displayed, it is possible to visually and intuitively recognize and accordingly easily know who has paid special attention to which utterance (talk), for example.

Further, in this embodiment, the conversation support system100includes the server5that stores voice data (and text data) of utterances, information on marking motions, information on the voice timeline, and information on the marking timeline.

This ensures a sufficient storage capacity regardless of performance and configuration of each terminal device1, and makes it possible to construct the conversation support system100that is more stable.

Modifications

In the above, an embodiment of the present disclosure has been described. It is needless to say, however, that the present invention is not limited to the embodiment, and can be modified in a variety of aspects within a range not departing from the scope of the present invention.

For example, in the above embodiment, as shown inFIG.1, the conversation support system100is applied to the case where the participants A to F are present in the conference rooms R1, R2and so forth, and bounce ideas off one another and utter. However, the conversation support system100is applicable to not only such a case but others.

For example, the conversation support system100is applicable to the case where only one person utters (speaks), such as the place of a speech.

In this case, although there is only one utterer (speaker) who delivers a speech, an audience who listens to the speech can participate in the place, namely in the conversation support system100, as performers who perform marking motions as needed by which markers are given to the content of the speech. In addition, the speaker who delivers the speech may also perform marking motions as needed by which markers are given to his/her own utterances.

Thus, even in the case where only one person utters, such as the place of a speech, the conversation support system100records, at all times, which talk the audience has valued and to which talk how the audience has reacted, without hindering the progress of the speech.

The conversation support system100does not need to have the server5.

When the conversation support system100does not have the server5, the controller11(hardware processor) of a terminal device1functions as the voice obtaining unit, the voice timeline creation unit, the marking analysis unit, the marking timeline creation unit, the data linkage unit, the utterer identification unit, the conversion-into-text unit and/or the like.

For example, when one terminal device1is provided in the conversation support system100, and participants participate in the utterance place from a conference room where the terminal device1is provided and/or from a place(s) where a microphone(s)2, a camera(s)3, a loudspeaker(s)4and/or the like connected to the terminal device1are provided, various data obtained/caught by all the microphones2, the cameras3and so forth provided in the conversation support system100are sent to the controller11of the terminal device1, and the controller11performs various processes.

As another example, when two or more terminal devices1are provided in the conversation support system100, one of the terminal devices1serves as a host. Various data input to the other terminal devices1are all sent to this host terminal device1, and the controller11of the host terminal device1functions as the voice obtaining unit, the voice timeline creation unit, the marking analysis unit, the marking timeline creation unit, the data linkage unit, the utterer identification unit, the conversion-into-text unit and/or the like.

In this case, the processing results of the controller11of the host terminal device1are sent to all the other terminal devices1in the conversation support system100to be shared. In this case, the terminal devices1may be connected to one another via the communication network6, such as Internet or a LAN, or with cables.

Further, the configuration of the conversation support system100is not limited to the one in which one controller functions as all the voice obtaining unit, the voice timeline creation unit, the marking analysis unit, the marking timeline creation unit, the data linkage unit, the utterer identification unit, the conversion-into-text unit and/or the like.

For example, the controller11of a terminal device1and the controller51of the server5may divide the roles therebetween and perform various processes.

Not making the controller51perform all the processes increases expectation of improvement in processing speed, and division of the roles between the terminal device1and the server5can reduce the load of the controller11, the storage12and/or the like of the terminal device1, so that the conversation support system100can operate smoothly.

Further, the detailed configuration and detailed operation of each component of the conversation support system100can be appropriately modified within a range not departing from the scope of the present invention.

Although one or more embodiments or the like of the present disclosure have been described and illustrated in detail, the disclosed embodiments or the like are made for purposes of not limitation but illustration and example only. The scope of the present invention should be interpreted by terms of the appended claims.