Patent Description:
Nowadays sophisticated speech synthesis algorithms are available on the market, which has opened a window for criminals or deceivers to leverage speech synthesis technologies such as Tacotron from Google to artificially fake voices with an arbitrary speaker and extreme quality up to indistinguishable from voices of a real human being. Using artificially faked voices represents a real danger to perform social engineering attacks through telephone on companies.

<CIT> discloses an apparatus that includes a microphone and a processor. The processor is configured to receive, via the microphone, audio comprising voice of a person, and determine whether the received audio is an actual voice or a synthesized voice. The apparatus also provides a first notification indicating that the received audio is the actual voice an a second notification indicating that the received audio is the synthesized voice.

<CIT> discloses a method and an apparatus for detecting synthesized speech. The method and apparatus include extracting a plurality of speech features from multiple segments of the speech signal, analyzing the plurality of speech features to determine whether the plurality of speech features exhibit periodic variation behavior and determining whether the speech signal is a synthetic speech signal or a natural speech signal on whether or not a periodic variation behavior of the plurality of speech features is detected.

Therefore, it is an object of the present invention to provide countermeasures against the threat of using artificially faked voices generated by a speech synthesized algorithm.

The technical problem is solved by the features of independent claims <NUM> and <NUM>.

The term "speech-based system" preferably defines a system that may be configured to transmit speech signals from a source entity, e.g. a calling party, to a destination entity, e.g. a called party, and to process speech signals cutted in signal portions to detect synthesized or natural speech.

Exemplary embodiments are described and illustrated in conjunction with the accompanying drawings.

At first, an exemplary classification algorithm is considered, which can be integrated into speech-based systems as shown in <FIG>, for example. The classification algorithm, if executed by a data processing unit, determines or infers whether a speech signal destined for a destination entity, for example a called party of a telecommunication system, is a synthetic or natural speech signal.

The classification algorithm may be trained by performing a supervised machine learning as is known from the artificial network technology. In particular, the classification algorithm is trained by using a plurality of first training sets of data, wherein each first training set of data represents a portion of a known synthesized speech signal and by using a plurality of second training sets of data, wherein each second training set of data represents a portion of a known natural speech signal. Each signal portion has a predetermined duration, which takes for example <NUM> second. With other words, each first training set of data may represent a synthesized speech snippet, wherein each second training set of data may represent a natural speech snippet of a human being.

It is to be noted, that each first and each second training set of data can represent the same speech features. For example a mel-frequency cepstrum can be used as the common speech feature. In this case, the speech signals to be checked are converted by executing a time-frequency transformation.

A supervised machine learning has the aim to train the classification algorithm such that it is able to evaluate a first predefined index value, whenever one of the plurality of the first training sets of data is used, thereby indicating a synthetic speech, and to evaluate a second predefined index value, whenever one of the plurality of second training sets of data is used, thereby indicating a natural speech. For example the first predefined index value is <NUM>, wherein the second predefined index value is <NUM>.

The classification algorithm can be implemented as a multilayer feedforward neural network and in particular by a two-layer feedforward neural network, wherein each layer comprises a plurality of artificial neurons, for example <NUM> artificial neurons.

A plurality of different applications are conceivable, which may use the trained classification algorithm to determine whether a speech signal is a synthetic or natural speech signal.

Now, different exemplary speech-based systems are illustrated, which are configured to may execute the trained classification algorithm for automatically detecting synthesized or natural speech for example during a telephone conversation between a calling party and a called party connected to a telecommunication network.

Referring now to <FIG>, an exemplary speech-based system <NUM> not covered by the claims is illustrated which comprises for example an IP-based telecommunication network <NUM> using the session initiation protocol (SIP) for establishing, handling and terminating IP-connections through the telecommunication network <NUM>.

In the following, only IP-telephone calls are considered. It is to be noted, that the speech-based system <NUM> is not limited to the use of IP-based or digital telecommunication networks. Other telecommunication systems can be used for transmitting analog speech signals. Also, the IP-based telecommunication network <NUM> can be a fixed or a wireless telecommunication network.

Furthermore, the exemplary speech-based system <NUM> comprises at least one communication apparatus <NUM>. Only for the sake of illustration, a single communication apparatus <NUM> is shown. It is to be noted, that the communication apparatus <NUM> may be implemented as a user telephone or a telephone system like a private branch exchange system or a smart home appliance. Only by way of example, it is assumed, that the communication apparatus <NUM> is a user telephone, for example an IP-capable fixed telephone. The IP-capable telephone <NUM> comprises a communication interface <NUM>, which may be configured to connect the telephone <NUM> via a DSL link to the IP-based telecommunication network <NUM>.

The communication device <NUM> further comprises a data processing unit <NUM>, which may be implemented as a microcontroller. Furthermore, the communication device <NUM> includes a memory or storage medium <NUM>, which stores a threshold value, the trained classification algorithm, instructions, which cause the data processing unit <NUM> to divide up incoming speech signals received at the communication interface <NUM> into signal portions, each portion having a predetermined duration. Preferably, the communication interface <NUM> comprises a physical network interface. The threshold value depends on the first and second predetermined index value, used by the classification algorithm during its training and can be set for example to <NUM>. This means that, if an index value evaluated by the classification algorithm is equal or greater than the threshold value, the speech signals inferred by the classification algorithm belong to natural speech signals. Furthermore, a firmware may also be stored in the memory <NUM>, which may cause the microcontroller <NUM> to operate, and control and monitor the communication device <NUM> in a usual manner. Moreover, the communication device <NUM> may include an output device <NUM>, for example a display.

In addition, the speech-based system <NUM> may comprise a communication device <NUM>, which is used later on to establish a telephone connection via the IP-based telecommunication network <NUM> to the telephone <NUM>. Only for the sake of explanation, it is assumed, that the communication device <NUM> can be used for generating synthetic speech as artificially faked voices intended for transmission via the IP-based telecommunication network <NUM> to the telephone <NUM>.

Now, the operation of the exemplary speech-based system <NUM> is explained.

It is assumed, that a telecommunication connection <NUM> has been established from the communication device <NUM> to the telephone <NUM> using, for example, the session initiation protocol and that artificially faked voice, i.e. synthesized speech, is generated by the communication device <NUM> and transmitted as IP-packets via the IP-based telecommunication network <NUM> to the telephone device <NUM> working as the called party. The incoming IP-packets carrying the speech signals are received at the communication interface <NUM> and delivered to the data processing unit <NUM> and for example outputted by a loudspeaker (not shown) of the communication device <NUM> to a user. The data processing unit <NUM> is configured to read out the instructions from the memory <NUM>, which causes the data processing unit <NUM> to divide up the incoming speech signals into portions of predetermined duration, also called snippets. For example, the incoming speech signals are cut into <NUM> second snippets. With other words: Each <NUM> second snippet comprises speech data of n IP-packets covering a speech signal duration of <NUM> second. The portions generated by the microcontroller <NUM> can be stored in the memory <NUM> or in any other storage medium implemented in the communication device <NUM>. Later on, the data processing unit <NUM> evaluates an index value by executing the trained classification algorithm stored in the memory <NUM> on at least one of the portions, and determines or infers in dependence of the first and second predefined index value and in dependence of the index value evaluated, whether the at least one portion belongs to a synthetic speech signal or a natural speech signal. In particular, the determination step is executed by the data processing unit <NUM> by comparing the evaluated index value with the threshold value stored in the memory <NUM>. If for example the evaluated index value is greater or equal than the threshold value the analyzed incoming speech signals are natural speech signals. However, in the present case, the evaluated index value is lower than the threshold value <NUM>, thereby indicating that the incoming and processed speech signals are synthesized speech signals. The threshold value may also be called a scalar danger value.

In order to may inform the user of the telephone <NUM> that the incoming speech signals is an artificially faked voice, the data processing unit <NUM> may be configured to generate an alert signal, if, as mentioned above, the evaluated index value is lower than the threshold value. In this case, the data processing unit <NUM> delivers an alert signal to the output device <NUM>, which is configured to output the alert signal in an optical and/or acoustic form.

Although the communication apparatus <NUM> has been described as a telephone, it can be for example implemented as a smart phone, smart home appliances like voice-activated door locks or any other voice driven systems, which are open to be deceived by synthetic voices.

It should be noted, that, if incoming speech signals received at the communication interface <NUM> of the communication device <NUM> are not digital, but analog signals, an A/D converter can be implemented in the communication apparatus <NUM>, which converts the analog speech signals into respective digital speech signals, which can be processed later on by the data processing unit <NUM> as described above.

Referring now to <FIG>, an exemplary speech-based system <NUM> is illustrated. Similar to the system <NUM> of <FIG>, the speech-based system <NUM> may comprise a telecommunication network, for example an IP-based telecommunication network <NUM>, which is implemented and configured to transmit for example digital speech signals in IP-packets from a communication device <NUM> to a communication apparatus <NUM>.

Similar to communication apparatus <NUM> the communication apparatus <NUM> may be a user telephone or a telephone system located at a customer. In the present case, the communication apparatus <NUM> is implemented as a telephone system, wherein a user telephone <NUM> is connected to the telephone system <NUM>. Of course, a plurality of communication apparatus <NUM> can be connected to the IP-based telecommunication network <NUM>. The communication apparatus <NUM> can be connected in a known manner via a communication interface <NUM> and via a DSL link to the telecommunication network <NUM>.

In addition, the speech-based system <NUM> comprises a computational system <NUM>, which, for example, can be integrated into the telecommunication network <NUM>, as is shown in <FIG>. It is noted, that the computational system <NUM> can be operated as an inferencing system.

The communication apparatus <NUM> is configured to receive IP-packets carrying the speech signals at its communication interface <NUM>, which are transmitted from the communication device <NUM> over the IP-based telecommunication network <NUM>. In particular, the communication apparatus <NUM> includes a data processing unit <NUM> and a memory <NUM>, which stores instructions which cause the data processing unit <NUM> to divide up the incoming speech signals received at communication interface <NUM> into portions or snippets of predetermined duration. Similar to data processing unit <NUM>, the data processing unit <NUM> can be configured to cut the incoming speech signals into a plurality of <NUM> second snippets. Furthermore, the data processing unit <NUM> and the communication apparatus <NUM>, respectively, are configured to transmit the <NUM> second portions to the computational system <NUM> using the IP-address of the computational system <NUM>. The data processing unit <NUM>, which is configured to divide up incoming speech signals into portions and the computational system <NUM> can be considered as an inferencing system for detecting synthesized or natural speech.

The dashed line, shown in <FIG>, indicates the transmission of the speech portions via communication interface <NUM> to computational system <NUM>.

As shown in <FIG>, the computational system <NUM> comprises a memory <NUM> configured to store similar to memory <NUM> a threshold value and the trained classification algorithm. A communication interface <NUM> is configured to receive in particular IP-packets carrying the signal portions from the communication apparatus <NUM>. The data processing unit <NUM> is configured to evaluate an index value by executing the classification algorithm stored in the memory <NUM> on at least one of the received portions, and furthermore to determine in dependence of the first and second predefined index value and in dependence of the index value evaluated, whether the at least one received portion belongs to a synthesized speech signal or to a natural speech signal. In particular, the determination step is executed by the data processing unit <NUM> by comparing the evaluated index value with the threshold value stored in the memory <NUM>. If for example the evaluated index value is greater or equal than the threshold value the analyzed signal portions received form the communication apparatus <NUM> belong to natural speech signals. Otherwise, the analyzed signal portions belong to synthesized speech signals. The threshold value stored in memory <NUM> may be for example <NUM>.

Now the operation of the speech-based system <NUM> is explained in detail.

Again, it is assumed that a telecommunication connection <NUM> has been established from the communication device <NUM> to telephone system as the communication apparatus <NUM> using, for example, the session initiation protocol. Furthermore, it is assumed that artificially faked voice, i.e. synthesized speech, are generated by the communication device <NUM> and transmitted in IP-packets via the IP-based telecommunication network <NUM> to the telephone system <NUM>, which is adapted to forward the IP-packets and the speech signals, respectively, to the telephone <NUM>, working as the called party.

The incoming speech signals received at the communication interface <NUM> are delivered to the data processing unit <NUM> and for example outputted by a loudspeaker (not shown) of the telephone <NUM>. The data processing unit <NUM> is configured to read out the instructions from the memory <NUM>, which causes the data processing unit <NUM> to divide up the incoming speech signals into portions of predetermined duration, also called snippets. For example, the incoming speech signals are cut into <NUM> second snippets. With other words: Each <NUM> second snippet comprises speech data of n IP-packets covering a speech signal duration of <NUM> second. Later on, the portions generated by the microcontroller <NUM> are transmitted to the communication interface <NUM> of the computational system <NUM>.

Now, the data processing unit <NUM> evaluates an index value by executing the classification algorithm, stored in memory <NUM>, on at least one of the portions received from communication apparatus <NUM> and determines in dependence of the first and second predefined index value and in dependence of the index value evaluated, whether the at least one portions belongs to a synthesized speech signal or to a natural speech signal. In the present case, it is assumed, that the data processing unit <NUM> has evaluated an index value lower than the threshold value, indicating that the speech signals transmitted from the communication device <NUM> to the communication apparatus <NUM> are synthesized speech signals.

In order to notify the user of the telephone <NUM> and the telephone system <NUM> that the ongoing speech conversation with the communication device <NUM> may be a fake conversation, the data processing unit <NUM> of the computational system <NUM> can be configured to generate an alert signal, if the evaluated index value is lower than the threshold value. In this case, the computational system <NUM> is configured to transmit the alert signal, as shown by the dashed line in <FIG>, to the communication apparatus <NUM> which can be configured, to output the received alert signal in an optical and/or acoustic form, for example on an output device <NUM>.

Simultaneously or alternatively the communication apparatus <NUM> can be configured to forward the alert signal received from the computational system <NUM> to the user telephone <NUM> which in turn can be configured to output the alert signal in an optical and/or acoustic way for example on a display <NUM>.

It should to be noted, that, although the computational system <NUM> is integrated into the telecommunication network <NUM>, it can be implemented as a cloud system which is connectable to the telecommunication network <NUM>.

Similar to communication apparatus <NUM> of <FIG>, the communication apparatus <NUM> can be a smart phone, a smart home appliance like a voice-activated door locks or other voice driven systems configured to cut incoming speech signals into signal portions of predetermined duration and to transmit these portions to the computational system <NUM> for further processing.

It is noted, that the communication apparatus <NUM> and the computational system <NUM> can be considered as two separated devices collaborating as an inferencing system to detect in particular malicious synthetic speech during a telephone conversation between telephone <NUM> and communication device <NUM>.

Referring now to <FIG>, a further exemplary speech-based system <NUM> is depicted.

Similar to the system <NUM> of <FIG>, speech-based system <NUM> may comprise a telecommunication network, for example an IP-based telecommunication network <NUM>, which is implemented and configured to transmit for example digital speech signals in IP-packets from a communication device <NUM> to a communication apparatus <NUM>. For the sake of illustration, it is assumed, that the communication device <NUM> can be used for generating outgoing artificially faked voices, i.e. malicious synthesized speech signals.

Similar to communication apparatus <NUM> the communication apparatus <NUM> may be a user telephone or a telephone system, for example a private branch exchange system or a smart home appliance, located at a customer or user. In the present case, the communication apparatus <NUM> is implemented by a telephone system, wherein a user telephone <NUM> can be connected to the telephone system <NUM>. Of course, a plurality of communication apparatus <NUM> can be connected to the IP-based telecommunication network <NUM>.

The communication apparatus <NUM> includes a communication interface <NUM>, which can be connected, for example via a DSL line to the telecommunication network <NUM>. The communication interface <NUM> is configured to receive incoming speech signals transmitted in IP-packets for example from the communication device <NUM> over the telecommunication network <NUM>. The communication interface <NUM> may comprise a physical network interface.

Furthermore, the exemplary speech-based system <NUM> comprises a computational system <NUM>, which can be integrated into a cloud system <NUM>. The computational system comprises a data processing unit <NUM> and a memory <NUM> which can store a threshold value, the trained classification algorithm, instructions causing the data processing unit <NUM> to divide up speech signals received at a communication interface <NUM>. For example, the speech signals may be generated and transmitted by the communication device <NUM> and may be destined for the communication apparatus <NUM>. Communication apparatus <NUM> includes a data processing unit <NUM>, which is configured to monitor, control and operate the communication apparatus in a known manner.

Now the operation of the exemplary speech-based system <NUM> is described in connection with <FIG>.

It is assumed, that communication device <NUM> has generated faked synthesized outgoing speech signals destined for the communication apparatus <NUM>. In a preferred embodiment, the communication device <NUM> uses the SIP protocol to establish a connection via the telecommunication network <NUM> to the communication apparatus <NUM> and the telephone <NUM>, respectively.

Now, there are, for example, two possibilities to deliver the outgoing speech signals from the communication device <NUM> to the cloud-based computational system <NUM>. One way is to transmit the outgoing speech signals from communication <NUM> via the telecommunication network <NUM> to the communication apparatus <NUM>, which in turn can be configured to forward the incoming speech signals to the telephone <NUM> and simultaneously forward the incoming speech signals via the telecommunication network <NUM> to the computational system <NUM> using its IP-address. In this case, the communication apparatus <NUM> knows the IP address of the computational system <NUM>.

In a second way, the telecommunication network <NUM> can be configured to detect, for example, speech signals coming from communication device <NUM> and to decide in dependence of the called destination, e.g. the communication apparatus <NUM> and the telephone <NUM>, respectively, that the speech signals destined for the telephone <NUM> are to be transmitted to the communication interface <NUM> of the communication apparatus <NUM> and also to the communication interface <NUM> of the cloud-based computational system <NUM>.

In any case, once IP-packets carrying speech signals have been received at communication interface <NUM>, the speech signals are delivered to the data processing unit <NUM>, which is caused by the instructions and the classification algorithm stored in memory <NUM>.

In particular, the determination step is executed by the data processing unit <NUM> by comparing the evaluated index value with the threshold value stored in the memory <NUM>. If for example the evaluated index value is greater or equal than the threshold value, e.g. <NUM>, the analyzed incoming speech signals received at the communication interface <NUM> are natural speech signals. However, in the present case, the evaluated index value is lower than the threshold value <NUM>, thereby indicating that the incoming and analyzed speech signals are malicious synthesized speech signals.

In order to notify the user of the telephone <NUM> and the telephone system <NUM>, that the ongoing speech conversation with the communication device <NUM> may be a fake conversation, the data processing unit <NUM> of the computational system <NUM> is configured to generate an alert signal, if the evaluated index value is lower than the threshold value stored in memory <NUM>.

Now the computational system <NUM> transmits the alert signal to the communication apparatus <NUM> via the telecommunication network <NUM> as depicted by the dashed line in <FIG>. The alert signal is received by the communication interface <NUM> and can be output in optical and/or acoustic form for example on an output device <NUM> of the communication apparatus <NUM> and or forwarded to the telephone <NUM>, which may be adapted to output the alert signal in an optical and/or acoustic form via an output device <NUM>.

It is noted, that the computational system <NUM> can be alternatively integrated into the telecommunication network <NUM>, as shown in <FIG> with respect to the computational system <NUM>.

Claim 1:
A speech-based system (<NUM>) for automatically detecting synthesized or natural speech, comprising
- a telecommunication network (<NUM>),
- at least one communication apparatus (<NUM>) connected to the telecommunication network (<NUM>) and configured to receive speech signals from a further communication device (<NUM>), the speech signals being transmitted over the telecommunication network (<NUM>),
- a computational system (<NUM>) comprising the following features:
- a memory (<NUM>) configured to store a classification algorithm, which has been trained by using a plurality of first training sets of data, each first training set of data representing a portion of a synthesized speech signal, each portion having a predetermined duration, and a plurality of second training sets of data, each second training set of data representing a portion of a natural speech signal, each portion having a predetermined duration, wherein the classification algorithm evaluates a first predefined index value, whenever one of the plurality of first training sets of data is used thereby indicating a synthesized speech, and wherein the classification algorithm evaluates a second predefined index value whenever one of the plurality of second training sets of data is used thereby indicating a natural speech,
- a communication interface (<NUM>) configured to receive speech signals addressed to the at least one communication apparatus (<NUM>),
- a data processing unit (<NUM>) configured to
i) divide up the speech signals received at the communication interface (<NUM>) into portions, each portion having a predetermined duration,
ii) to evaluate an index value by executing the classification algorithm stored in the memory (<NUM>) on at least one of the portions, and
iii) to determine in dependence of the first and second predefined index value and in dependence of the index value evaluated, whether the at least one portion belongs to a synthesized speech signal or to a natural speech signal.