System and method for providing feedback to a caller in response to the modulation of an incoming voice communication

In order to facilitate the modulation of speech delivery to a participant that is driving or otherwise performing a parallel task, a feedback mechanism is incorporated in order to inform the remote participant of the progress of the delivery of his/her speech at their end of the conversation. In this way, the remote participant is both aware of the modulation occurring at the other end and can adjust their speech to adapt to the current conditions, using the feedback as a guide.

TECHNICAL FIELD

The following relates to systems and methods for providing feedback to callers in response to the modulation of an incoming voice communication.

BACKGROUND

Using portable devices while driving can be dangerous because such use can be distracting and can lower reaction times for the driver. This practice is particularly applicable to conversing on a mobile phone, whether or not the mobile phone has a “hands-free” capability. Nevertheless, even when laws forbid using a mobile phone while driving, a hands-free option is typically allowed. Despite freeing up the driver's hands, participating in a conversation while driving can still be distracting and can create potentially dangerous conditions for both the driver and others on the road.

DETAILED DESCRIPTION OF THE DRAWINGS

In order to reduce distractions caused by engaging in a conversation (i.e. voice communication) using a mobile device, e.g. while driving or performing some other parallel task, the rate of speech delivery being provided to the recipient can be modulated such that it is better adapted to the immediate conditions facing the recipient. For example, it has been observed that when engaged in a conversation with another passenger in a vehicle, one or more of the participants in the conversation often slow down their speech or introduce pauses or delays while they are speaking in response to driving conditions such as bad weather, heavy traffic, high speeds, etc. The participants may even stop talking to one another when driving conditions are particularly demanding such as when slowing down, turning, or sudden or abrupt stops are required.

When conversing with another participant who is not in the vehicle, e.g. via a telephone call using a mobile device, the remote participant (the one that is not in the vehicle) typically does not react to the immediate driving conditions for at least the reason that they are not present to observe such conditions. As such, the conversation can become potentially dangerous to the participant in the vehicle.

The rate of delivery of speech to the participant in the vehicle can be modulated artificially, e.g. by detecting external events such as speed, acceleration, deceleration, light conditions, weather conditions, traffic conditions, time of day, location, turning and/or navigating an intersection, among many others. Mechanisms such as a GPS receiver can also be used to distinguish between different conditions, for example whether a driver is following a curve in a roadway versus making a turn at an intersection, in order to determine if the speech should be modified. The monitoring of events enables the immediate driving conditions to dictate the rate at which the participant in the vehicle is provided with an incoming voice signal from another participant. If the rate of speech or voice signal delivery is modified for the participant in the vehicle, the remote participant may be unaware that the delivery of their speech is being modulated and by how much. As a result, the remote participant may become frustrated due to the delay in receiving a reply from the participant in the vehicle and may even believe the connection is bad. Similarly, without being able to react to the introduction of a modulation of their speech at the other end, the remote participant may continue to speak at a rate that does not allow the other end to re-synchronize or “catch up”.

In order to facilitate the modulation of speech delivery to a participant that is driving or otherwise performing a parallel task, a feedback mechanism can be incorporated in order to inform the remote participant of the progress of the delivery of his/her speech at their end of the conversation. In this way, the remote participant is both aware of the modulation occurring at the other end and can adjust their speech to adapt to the current conditions, using the feedback as a guide. As will be explained in greater detail below, the feedback can be provided in various forms, including tactile feedback (e.g. vibration or other discernible movements), audible feedback, visual feedback, or any combination thereof.

Turning now toFIG. 1, a pictorial view of the use of a mobile device10in a moving vehicle2is shown. In this example, two vehicles2are shown moving in opposite directions on a road4, with one of the vehicles2carrying a mobile device10that is participating in a voice communication with a remote participant and its remote device via a wireless network12. It can be seen that the remote device can be another mobile device6or a landline telephone8connectable to the wireless network12via a plain old telephone service (POTS)14as is well known in the art and thus the remote device can be any telecommunications device configured for participating in voice communications.

FIG. 2illustrates a voice/speech waveform16that has been subjected to one example speech modulation technique, namely wherein the normal speech18is delayed or paused for a period of time, hereinafter referred to as a “delay20”. This technique effectively breaks up a speech waveform to provide pauses to the recipient participant for managing the delivery of speech according to one or more factors, criteria or external event as discussed by way of example below. It can be appreciated that the delays20may be used to mimic natural pauses that a participant may utilize when modulating their speech according to their environment (e.g. difficult driving conditions).

FIG. 3illustrates a speech waveform16that has been subjected to another example speech modulation technique, namely wherein the normal speech18is modified rather than interrupted. In this case the rate of speech delivery is decreased, thus providing a portion of the speech at a slower rate, hereinafter referred to as a “decreased rate change22”.

It can be appreciated that the two techniques shown inFIGS. 2 and 3can also be combined such that the normal speech18is modified to include both delays20and rate changes22as shown inFIG. 4.

It can also be appreciated that when slowing down or pausing speech delivery, there may be a need to re-synchronize or “catch up” the remote participant's speech at some point. Natural pauses in the conversation can allow for such re-synchronization, however, in other instances, it may be advantageous to artificially catch up the speech by speeding up the delivery at an appropriate time. As shown inFIG. 5, the signal16can thus include an increased rate change23. In one example scenario, the external events can be used to detect that the mobile device10is stopped at a red light or has pulled over, etc. which would be an appropriate time to begin accelerating the speech delivery in order to “catch up”. This can be particularly useful in situations where the conversation is relatively one-sided, such as when the mobile participant is using their mobile device10to listen in on a conference call or is not continuously engaged in, e.g., a multi-participant conversation.

In order to control the delivery of speech to the remote device, as shown inFIG. 6, a modulator30can be used. In this example, each of two participants, Participant A and Participant B, communicate with each other in a voice communication via data communication modules24. Each data communication module24can obtain voice inputs via a microphone26and can provide voice outputs (speech delivery) via a speaker28as is well known in the art. It is assumed inFIG. 6that Participant B is communicating while driving or otherwise performing a parallel task and thus should have the delivery of speech to them controlled. As such, the data communication module24at Participant B uses a modulator30to control outputs to its speaker28and to generate feedback36(see alsoFIG. 7) to be sent to Participant A. As seen inFIG. 6, the modulator30may comprise a feedback module32that is used to generate the feedback36. Although shown as part of the modulator30inFIG. 6it can be appreciated that the feedback module32may also be a separate component or may be an integral part of the modulator30not requiring delineation therefrom. As such, it can also be appreciated that the feedback module32and modulator30can be separate and distinct components in some embodiments wherein the feedback module32operates independently from the modulator30by determining what if any modulation is being applied using the modulator30.FIG. 6also shows Participant A having a feedback module32to illustrate that the feedback36can be generated by Participant B and delivered via the voice channel or can be generated at Participant A's device6,8. For example, Participant B could send out-of-band information using another communication channel that is converted into the feedback36at the other end. As such, it can be appreciated that the feedback36can be delivered or otherwise provided to the remote device using any suitable delivery mechanism.

FIG. 7provides one example configuration for the modulator30shown inFIG. 6. The modulator30in this embodiment provides a delay module34to intercept normal incoming speech signals18and references or otherwise relies on one or more external events to determine an appropriate delay20or rate change22to be applied to the signal18. In this example, a rate change22is applied to slow down the speech at that particular time. The rate changed22output is then provided to the speaker28to be output to Participant B. The modulator30in this configuration also comprises a feedback module32which is controlled according to the delay20or rate changes22,23applied to the incoming signal. Based on the way in which the incoming signal is modified, a “modulation factor” can be provided to the feedback module32and the feedback module32can use this modulation factor in order to generate feedback36to be sent to Participant A to indicate to them how their speech is being modified at the other end. It can be seen inFIG. 7that in this embodiment, the normal speech signals18picked up by the microphone26at Participant B are still able to be communicated back to Participant A. Therefore, in this example, the feedback36is provided in conjunction with the voice signal18from Participant B and typically does not substantially interfere with or mask the voice signal18. It can be appreciated however that in other embodiments, the feedback module32can intercept the voice signals18being sent back to Participant A in order to interleave or otherwise separate the feedback36from the voice signal18.

The feedback36can take various forms, examples of which are shown inFIG. 8.FIG. 8illustrates an example modulated signal16which, for ease of illustration, comprises a number of delays20introduced throughout the signal16.FIG. 8also illustrates a signal that corresponds to the modulated signal16, wherein a feedback interval38coincides with a delay20in the signal16. InFIG. 8, the feedback36is either provided during the feedback interval38, or not provided otherwise, and thus resembles a pulse-width-modulated (PWM) signal. The form that the feedback36takes can vary, e.g. background music, beeps, visual output (progress bars, waveform output, etc.), etc. As such, each feedback interval38of the signal16shown inFIG. 8represents a period of time in which the feedback36provides a distinguishable alert, notification, cue or other output to the remote device (i.e. Participant A in this example) to indicate to them that their speech signal16is being modulated.

Although the first example shown inFIG. 8illustrates a constant signal within each feedback interval38, the feedback36itself can vary during that interval38.FIG. 8also illustrates one example in which the feedback36during an interval38adecreases in magnitude (e.g. volume decreases, colour changes, etc.) at a particular rate of change.FIG. 8also illustrates feedback36which decreases in magnitude during an interval38bin a stepped fashion. The stepped feedback interval38bcan be used to indicate a transition within the feedback interval38b, e.g. as the delay20comes to an end.FIG. 8also illustrates feedback36which decreases logarithmically, e.g. such that an audible feedback “fades” as a feedback interval38ccomes to an end.FIG. 8also shown a feedback interval36dthat provides constant feedback36throughout that interval36d. It can be appreciated that the different types of feedback intervals38,38a,38b,38c,38dare only examples and a variety are shown in the same signal for illustrative purposes only. For example, one type of feedback interval38can be used for each delay20or different ones can be chosen under different circumstances in various applications.

FIG. 8illustrates another feedback signal comprising feedback intervals38ethat provide discrete pulses within each interval38e. For example, the feedback interval38emay coincide with a series of beeps or tones. As can be seen, the frequency of such beeps or tones can vary to indicate to the remote device how much of their previous speech still needs to be delivered. In this example, the beeps slow down as the interval38ecomes to an end to signify that the remote device can begin to speak again. In this way, if delays20become long due to either poor driving conditions or due to rapid speech by the remote device, the beeps within the interval38ewill be more frequent for a longer period of time which can enable the remote device to react to the feedback36to allow the remote device to catch up.

It can be appreciated that by not only providing feedback36during the feedback intervals38but also varying the feedback36provided during the intervals38f, the remote device can adapt their speech delivery to accommodate changing conditions. In this way, if they begin to speed up their speech due to a diminishing feedback36, should conditions change, a new interval38fcan begin or the same interval38fcan be modified to increase the magnitude of the feedback36accordingly as shown inFIG. 8.

FIG. 9illustrates another example waveform16, which, in this example includes an decreased rate change22.FIG. 9also illustrates a feedback interval38gwherein the magnitude of the feedback36decreases logarithmically.FIG. 9also illustrates a feedback interval38hwhich includes a series of discrete beeps or tones that both decrease in magnitude and increase in frequency as the interval38hprogresses.FIG. 9also illustrates a feedback interval38iwhich includes a constant feedback36throughout. Therefore, it can be appreciated fromFIG. 9that the principles shown inFIG. 8can also be applied to speech waveforms that are modulated using a rate change22.

Turning now toFIG. 10, an example external view of a mobile device10is shown, the mobile device10displaying an interface40. InFIG. 10, the mobile device10is providing a tactile feedback (e.g. vibration) as indicated by the movement lines. Tactile feedback is particularly advantageous as it would not interfere with the audio/speech delivery of the ongoing conversation. Moreover, the remote participant would not be required to move their device away from their ear, for example to observe visual feedback. Instead, the tactile feedback would be felt through the user's hand.

As discussed above, the feedback36may also be an audible feedback in the form of background music, beeps or tones, pre-recorded messages indicative of an external event, and the like.FIG. 11illustrates a mobile device10delivering an audible feedback36. The audible feedback36may be particularly advantageous in hands-free systems wherein the display of the mobile device10may not be accessible.

As also discussed above, the feedback36may be visual in nature. InFIG. 12, the interface40includes or has overlaid thereon, a call modulation interface component42(e.g. a pop-up window as shown). The call modulation progress interface component42in this example includes a progress bar44which provides visual feedback36indicative of the progress of delivery of the remote participant's speech. The progress bar44can be displayed whenever a delay20or rate change22is required and during the corresponding feedback interval38in order to indicate to the remote participant, how much of their speech has been delivered and at what rate. It can be appreciated that the progress bar44can be displayed in combination with an audible, tactile, or any other type of feedback36, or can be displayed instead of such feedback36. For example, the progress bar44or other visual feedback36can be linked to a user's profile such that if they do not wish to receive audible notifications for other communications, they instead receive the visual feedback36. When used in combination, the interface component42can assist the user in identifying what event corresponds to the audible alert, which can be particularly advantageous when the mobile device10uses multiple audible alerts for multiple reasons.

FIG. 13illustrates another example of visual feedback36, wherein the interface component42includes a speech waveform46that resembles the actual progress of the speech as it is delivered to the recipient (mobile) participant. It can be appreciated that the examples shown inFIGS. 10 to 13are for illustrative purposes and various other visual cues or notifications can be used. For example, a blinking element can be used that mimics audible beeping or replaces such audible beeping. Also, a visual element that changes colour can also be used, e.g. from red to yellow to green to indicate when the remote participant should begin to speak again. Any combination of these elements can be incorporated into the feedback36and in some embodiments, such elements can be user configurable or set according to preferences. IT policies or other criteria. Moreover, as noted above and illustrated inFIG. 10, tactile feedback such as movement or vibration of the remote device used by the remote participant can also be used, either alone or in combination with one or more of the visual and audible feedback types.

FIG. 14illustrates a set of example computer executable instructions that may be performed by the modulator30in order to both modulate a voice signal on the mobile device10and provide feedback36to the remote participant (i.e. the other caller). At50, it is assumed in this example that Participant A initiates a call and a voice signal is sent to Participant B. The voice signal is obtained by the mobile device10for Participant B at52. In this example, one or more external events are detected at54which causes the modulator30to modulate the incoming signal at56, e.g. impose one or more delays20, one or more rate changes22,23or a combination of both. The modulated speech is then output by the modulator30at58. For example, tactile feedback36as illustrated inFIG. 10, audible feedback36as shown inFIG. 11to provide a series of beeps, background music, etc., or a visual output may be displayed as shown inFIGS. 12 and 13according to the waveforms shown inFIGS. 8 and 9. Once the signal is modulated at56, a modulation factor can also be determined at60in order to generate feedback for Participant A. For example, as shown inFIGS. 2 and 3, the length of the pause or delay20(denoted B) or the decrease in the rate of speech (denoted B′) can be used to determine the nature of the feedback36to be provided during one or more feedback intervals38. The feedback36is then generated at62and sent to the remote device (in this example the caller) at64. The feedback36is then received by Participant A at66and the feedback36output at68. It can be appreciated that Participant A may be using a mobile device10or another device such as a telephone8as discussed above.

After the feedback36is sent to Participant A, the modulator30then determines if the voice signal is still being modulated at70. If so, the modulator can return to60in order to continue to modify the feedback36or otherwise determine when the delay20or rate change22is no longer required. If the modulation is no longer being applied, the modulator's process ends at72.

Turning now toFIG. 15, an example set of computer executable instructions are illustrated that may be performed by Participant A's device in outputting the feedback36at68. For example, if Participant A is using a mobile device10that also has a modulator33or otherwise has some form of a feedback module33as shown inFIG. 6, the feedback module30or modulator30may initiate the modulator interface42at74, generate a representation of the feedback36at76(e.g. progress bar44or waveform46), and the representation is displayed at78, e.g. as shown inFIG. 12or13. It can be appreciated that similar operations may be performed for other types of feedback36. For example, audible feedback36may require initiation of a speaker module, the generation of the audible signal, and the output of this audible signal over the speaker28. Similarly, tactile feedback36may require initiation of a vibration system.

FIG. 16provides a block diagram of an example embodiment of a mobile device10. The mobile device10comprises a number of components such as a main processor102that controls the overall operation of the mobile device10. Communication functions, including data and voice communications, are performed through a communication subsystem104. The communication subsystem104receives messages from and sends messages to a wireless network12. In this example embodiment of the mobile device10, the communication subsystem104is configured in accordance with the GSM and GPRS standards, which are used worldwide. Other communication configurations that are equally applicable are the 3G and 4G networks discussed above. New standards are still being defined, but it is believed that they will have similarities to the network behaviour described herein, and it will also be understood by persons skilled in the art that the embodiments described herein are intended to use any other suitable standards that are developed in the future. The wireless link connecting the communication subsystem104with the wireless network12represents one or more different Radio Frequency (RF) channels, operating according to defined protocols specified for GSM/GPRS communications.

The main processor102may also interact with additional subsystems such as a Random Access Memory (RAM)106, a flash memory108, a display110, an auxiliary input/output (I/O) subsystem112, a data port114, a keyboard116, a speaker118, a microphone120, a GPS receiver121, short-range communications122, and other device subsystems124. As will be discussed below, the short-range communications122can implement any suitable or desirable device-to-device or peer-to-peer communications protocol capable of communicating at a relatively short range, e.g. directly from one device to another. Examples include Bluetooth®, ad-hoc WiFi, infrared, or any “long-range” protocol re-configured to utilize available short-range components. It will therefore be appreciated that short-range communications122may represent any hardware, software or combination of both that enable a communication protocol to be implemented between devices or entities in a short range scenario, such protocol being standard or proprietary.

It can be appreciated that the keyboard116may include a physical keyboard or a virtual or “soft” keyboard (not shown) which may be implemented, for example, by providing images of keys rendered on a touch screen, OLED, etc.

Some of the subsystems of the mobile device10perform communication-related functions, whereas other subsystems may provide “resident” or on-device functions. By way of example, the display110and the keyboard116may be used for both communication-related functions, such as entering a text message for transmission over the network12, and device-resident functions such as a calculator or task list.

The mobile device10can send and receive communication signals over the wireless network12after required network registration or activation procedures have been completed. Network access is associated with a subscriber or user of the mobile device10. To identify a subscriber, the mobile device10may use a subscriber module component or “smart card”126, such as a Subscriber Identity Module (SIM), a Removable User Identity Module (RUIM) and a Universal Subscriber Identity Module (USIM). In the example shown, a SIM/RUIM/USIM126is to be inserted into a SIM/RUIM/USIM interface128in order to communicate with a network. Without the component126, the mobile device10is not fully operational for communication with the wireless network12. Once the SIM/RUIM/USIM126is inserted into the SIM/RUIM/USIM interface128, it is coupled to the main processor102.

The mobile device10is typically a battery-powered device and in this example includes a battery interface132for receiving one or more rechargeable batteries130. In at least some embodiments, the battery130can be a smart battery with an embedded microprocessor. The battery interface132is coupled to a regulator (not shown), which assists the battery130in providing power V+ to the mobile device10. Although current technology makes use of a battery, future technologies such as micro fuel cells may provide the power to the mobile device10.

The mobile device10also includes an operating system134and software components136to146which are described in more detail below. The operating system134and the software components136to146that are executed by the main processor102are typically stored in a persistent store such as the flash memory108, which may alternatively be a read-only memory (ROM) or similar storage element (not shown). Those skilled in the art will appreciate that portions of the operating system134and the software components136to146, such as specific device applications, or parts thereof, may be temporarily loaded into a volatile store such as the RAM106. Other software components can also be included, as is well known to those skilled in the art.

The subset of software applications136that control basic device operations, including data and voice communication applications, may be installed on the mobile device10during its manufacture. Software applications may include a message application138, a device state module140, a Personal Information Manager (PIM)142, a connect module144and an IT policy module146. A message application138can be any suitable software program that allows a user of the mobile device10to send and receive electronic messages, wherein messages are typically stored in the flash memory108of the mobile device10. A device state module140provides persistence, i.e. the device state module140ensures that important device data is stored in persistent memory, such as the flash memory108, so that the data is not lost when the mobile device10is turned off or loses power. A PIM142includes functionality for organizing and controlling data items of interest to the user, such as, but not limited to, e-mail, text messages, instant messages, contacts, events, and voice mails, and may interact with the wireless network12. A connect module144implements the communication protocols that are required for the mobile device10to communicate with the wireless infrastructure and any host system25, such as an enterprise system, that the mobile device10is authorized to interface with. An IT policy module146receives IT policy data that encodes the IT policy, and may be responsible for organizing and securing rules such as the “Set Maximum Password Attempts” IT policy.

Other types of software applications or components139can also be installed on the mobile device10. These software applications139can be pre-installed applications (i.e. other than message application138) or third party applications, which are added after the manufacture of the mobile device10. Examples of third party applications include games, calculators, utilities, etc. The additional applications139can be loaded onto the mobile device10through at least one of the wireless network12, the auxiliary I/O subsystem112, the data port114, the short-range communications subsystem122, or any other suitable device subsystem124.

The data port114can be any suitable port that enables data communication between the mobile device10and another computing device. The data port114can be a serial or a parallel port. In some instances, the data port114can be a USB port that includes data lines for data transfer and a supply line that can provide a charging current to charge the battery130of the mobile device10.

For voice communications, received signals are output to the speaker118, and signals for transmission are generated by the microphone120. Although voice or audio signal output is accomplished primarily through the speaker118, the display110can also be used to provide additional information such as the identity of a calling party, duration of a voice call, or other voice call related information.

For composing data items, such as e-mail messages, for example, a user or subscriber could use a touch-sensitive overlay (not shown) on the display110that is part of a touch screen display (not shown), in addition to possibly the auxiliary I/O subsystem112. The auxiliary I/O subsystem112may include devices such as: a mouse, track ball, infrared fingerprint detector, optical trackpad, or a roller wheel with dynamic button pressing capability. A composed item may be transmitted over the wireless network12through the communication subsystem104.

Although the above principles have been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the scope of the claims appended hereto.