Patent Description:
Devices are becoming increasingly intelligent. As well as existing computing devices, cars, mobile phones and the like that already had user interfaces and computing capabilities, users are now being offered devices with integrated computers such as watches, coffee makers and door locks. Furthermore, new classes of devices such as smart speakers and other personal assistants are coming on to the market.

While capabilities of the various devices can vary significantly, the user interfaces offered are generally similar - they are typically based on buttons, touchscreen or voice-based inputs and provide visual and/or audible output.

As the number of devices carried by users and also those in the user's general environment increases, so too do the number of devices demanding attention. Information overload is a serious risk and a reason why the uptake in intelligent devices, particularly by those who are not used to the technology, has been met with reluctance. <CIT> describes a system that is configured to modify data generating haptic feedback to account for changes in user perception of haptic feedback.

According to an aspect of the present invention, there is provided a user interface system as claimed in claim <NUM>.

The user interface system may be configured to guide the user through a series of stimuli applied by the user interface device via the I/O unit and determine a degree of recognition by the user for each stimulus.

The controller may be configured to record degree of recognition by the user for each stimulus in the data repository.

The controller may be operable to determine the optimised set of stimuli to be used by the user interface device in dependence on the recorded degree of recognition for each stimulus in the data repository, the optimised set matching the number of stimuli provided by the user interface device.

The system may further comprise a GUI accessible by the user and operable to assign stimuli to functions of the user interface device.

The system may further comprise a training system arranged to cause, via the I/O unit, the user interface device to provide training on a stimulus or a set of stimuli to the user.

The system may further comprise a machine learning system, the machine learning system being configured to guide the selection of stimuli and/or optimised stimuli for the user to be trained upon.

The system may further comprise an Appstore, the appstore includes apps and/or extensions that can be selectively installed onto a user device, the system being configured to determine assignment of stimuli in the user device for apps installed via the appstore.

The system may further comprise a stimulus user profile in the data repository for each user, the stimulus user profile including data on the optimised sets of methods for stimulation for the user.

The stimuli may comprise one or more selected from a set including: vibration, electrical, stretching of the user's skin, pressure applied to the user's skin, sound and light.

The stimuli may be provided by an actuator that is controlled to provide a coded representation of a function, the representation being selected from one or more of a set including:
amplitude modulation, frequency modulation, spatiotemporal patterns across the multiple actuators.

The user interface system may be configured to guide the user through a series of stimuli applied by the user interface device via the I/O unit and determine a degree of reaction by the user for each stimulus, the controller being operable to determine the optimised set of stimuli to be used by the user interface device in dependence on the degree of reaction to the stimuli by the user.

The reaction may be a physical reaction, the system including one or more sensors configured to measure the user's physical reaction.

The reaction may be an emotional reaction, the user interface system being configured to receive an input from the user rating his or her emotional reaction to the stimulus, the user interface system being configured to determine the degree of reaction in dependence on the rating.

The user interface system may be configured to group stimuli in dependence on a user's reaction.

The user interface system may be configured to assign grouped stimuli to a common contact, contact type, role, demographic or other grouping relevant to the user.

According to another aspect of the present invention, there is provided a method as claimed in claim <NUM>. The method may include applying one or more selected stimuli to a user device.

Embodiments seek to provide a framework to organise and coordinate the outputs of devices to suit abilities of users. Embodiments of the present invention seek to provide a system and method in which extensibility of a user interface can be managed in dependence on user capabilities. Preferred embodiments seek to provide a user interface system in which a user can be trained and/or tested to improve recognition of differing inputs and/or identify areas of granularity of user inputs that cannot be differentiated by a user. In this manner, user capabilities can be tested and monitored. Embodiments furthermore seek to provide a user interface that can be customised and improved in dependence on the testing and training to best suit the user's capabilities. While the aim is not to improve the user's capabilities, there may be situations where this is achieved. Embodiments seek to improve the stimulus paradigm to suit the (perhaps improving) capabilities of the user.

For example, a user interface may have a haptic output device for providing outputs to a user - the system can test and optionally train the user so as to determine and optionally improve the range of haptic stimuli the user can recognise from the device. Once the range of recognisable stimuli is identified, the system can associate outputs such as differing alert events with different haptic stimulus outputs by the device that have been determined to be distinguishable by the user. Optionally, the user may undergo training to improve his or her ability to recognise haptic stimuli so as to increase the number of stimuli that can be provided by the device. Optionally, the user interface device may be the output device for an extensible computing device (such as a mobile phone, smart watch etc on which new apps may be installed) - the system can optionally determine an optimised set of stimuli for the installed functions of the computing device in dependence on the user's capabilities. Where a threshold of detectable stimuli from the output device is reached, the system may be configured to suggest training and/or limitation of installed functions to ensure the user is able to successfully recognise the assigned stimuli.

Preferred embodiments seek to support the creation and maintenance of a haptic (or other stimulus type) user profile containing the optimised sets of methods for stimulation for that user. The profile can, in turn, be used to determine best configuration of user interfaces (apps etc) or devices as they are acquired by the user. The profile can also be used to determine when the stimuli used in an existing device could be changed or expanded upon.

Preferred embodiments also seek to provide a system in which multiple user interfaces and/or user interface devices are associated with a user profile such that common alerts/events across different devices have common stimuli associated with them and different alerts, events etc are associated with different stimuli. In this way, embodiments seek to optimise and coordinate the output of multiple devices and have a unified set of protocols for an individual user that is applied to all the devices of that user.

Embodiments of the present invention will now be described by way of example only with reference to the accompanying drawings in which:.

Embodiments of the present invention are particularly suited to managing and supporting user interfaces that interact via haptic stimuli. However, it will be appreciated that embodiments may support actuators that provide (by themselves or in combination) multiple stimuli types and/or stimuli other than haptic.

Vibrotactile stimulation, or haptics, has been widely used as a mechanism for notification. In recent years, uses have expanded considerably.

Examples of how haptics can be used are set out below.

To-date, haptic stimulators have relied on simple binary activation using a single frequency. The spatial resolution of the human tactile sense, in most parts of the body is not very high. As a result, only simple actuators and haptic stimuli have been used.

Embodiments of the present invention enable more complex actuators to be employed, leading to more complex and varied stimulation patterns. They could, for example, use a combination of amplitude and frequency modulation and/or employ multiple spaced actuators to provide stimuli formed from defined spatiotemporal patterns across the multiple actuators. In this case, varying the spatial pattern of activation across multiple actuators in a specific spatial arrangement (e.g. a row of actuators on a bracelet or belt) can be used to encode information.

In addition to haptic stimulators, embodiments of the present invention may be used with other user interface devices that provide stimuli to users to indicate alerts, action, events and the like. Such stimulators may include:.

Although the following discussion focuses on haptic stimuli, it will be appreciated that others such as those above could also be used in addition or in alternative.

<FIG> is a schematic diagram of a user interface system according to an embodiment.

The user interface system <NUM> includes a controller <NUM>, a data repository <NUM> and an I/O unit <NUM>.

The user interface system <NUM> is configured to cause, via the I/O unit <NUM>, a user interface device <NUM> of a user <NUM> to provide a stimulus to the user <NUM>. The user interface system <NUM> is further configured to receive, from the user <NUM>, data on the stimulus. In response to the stimulus and the data provided by the user <NUM> on the stimulus, the user interface system <NUM> is arranged to record, in the data repository <NUM>, stimulus recognised by the user.

Preferably, under control of the controller <NUM>, the user interface system <NUM> is configured to guide the user through a series of stimuli applied by the user interface device <NUM> via the I/O unit <NUM> and determine a degree of recognition for each stimulus.

Preferably, the controller <NUM> is configured to record degree of recognition by the user <NUM> for each stimulus in the data repository <NUM>.

Preferably, the user interface device <NUM> is configurable, the controller <NUM> is operable to determine an optimised set of stimuli to be used by the user interface device <NUM> in dependence on the recorded degree of recognition for each stimulus in the data repository, the optimised set matching the number of stimuli provided by the user interface device <NUM>. The controller is arranged to communicate with the user interface device <NUM> and configure the user interface device <NUM> to use the optimised set of stimuli for its functions. The user interface system <NUM> may include a GUI accessible by the user <NUM> to assign stimuli to particular functions of the user interface device <NUM>. It will be appreciated that systems other than GUIs could also be used including verbal and gesture based interfaces.

<FIG> is a schematic diagram of the user interface system of <FIG> highlighted selected preferred options.

Preferably, the user interface system <NUM> is configured to invoke a training system <NUM>, the training system <NUM> being arranged to cause, via the I/O unit <NUM>, the user interface device <NUM> to provide training on a stimulus or a set of stimuli to the user <NUM>. Training may, for example, include providing a number of different stimuli and guiding the user so as to highlight the sensation provided by the respective stimulus. Due to repeated stimuli, the user can be educated to recognise and differentiate different stimuli.

Preferably, the controller <NUM> and/or the training system <NUM> includes a machine learning system. The machine learning system may guide the selection of stimuli and/or optimised stimuli. It may also guide the selection of stimuli for the user to be trained upon (or those to be avoided as the user is not ready or able to differentiate).

<FIG> is a schematic diagram illustrating a further embodiment.

Preferably, the user interface system <NUM> includes an Appstore <NUM>. The Appstore <NUM> includes apps and/or extensions that can be selectively installed onto a user device.

The user device may be the user interface device <NUM> or it may be a user device <NUM> that communicates with the user via the user interface device <NUM>. For example, the user interface device <NUM> may be a smartwatch with a haptic stimulator. It will be appreciated that apps may be installable onto the smartwatch <NUM> or they may be installed onto a smartphone <NUM> linked to the smartwatch <NUM> via Bluetooth, Bluetooth Low Energy, ANT+ or similar technologies. In the latter case, the app could be installed on the smartphone <NUM> and provides stimulus to the user via the smartwatch <NUM>.

The user interface system <NUM> may furthermore guide the user on combinations of apps that can be accommodated. For example, where two apps want to use the same stimulus, the user interface system <NUM> may suggest changing the stimulus of one or the other. Similarly, where the user seeks to install more apps that provide stimuli exceeding those the user interface device <NUM> can provide or exceeding the number the user has shown he or she can recognise, the user interface system <NUM> may provide a warning, block installations or suggest more training to extend the stimuli recognition of the user.

A particular advantage of embodiments of the present invention is that they focus on flexibility and extensibility. The type and capability of the user interface device and its stimulator can vary and as more complex devices and stimulators become available or different ways of stimulating a user become recognised, embodiments can accommodate these. The user interface system is distinct and separable from any specific user interface device hardware.

Communication with the user interface device and its stimulator will depend on the particular device but would typically be via Bluetooth, an API or similar. In the case of smartphones and the like, an app may be installed on the smartphone to act as a gateway to the smartphone hardware for the user interface system <NUM>.

Embodiments seek to enable a single device (with one or more actuators) to provide pertinent haptic communication for multiple independent applications. Embodiments seek to achieve this by employing a unique combination of cognitive modelling with machine learning (to identify the stimuli the user can recognise) and complex temporal vibrotactile stimulation protocols (which can provide a granular and extensive vocabulary of stimuli for communicating with the user) to create a rich and complex language that is uniquely tailored and private to an individual user.

Preferred embodiments use structured stimulation patterns that do not rely on a spatial arrangement of actuators to be effective. To achieve this, combinations of distinct and varying rhythms with frequency and amplitude modulation are used to create rich structured stimulation patterns. The main challenge with this type of stimulation is that accuracy and precision are often variable between individuals but internally consistent within an individual. To solve this problem and create a language that is widely useable, a combination of modelling approaches and machine learning algorithms are used to personalise and adapt structured haptic stimulation to create a personalised and efficient haptic language system for transmitting information.

Embodiments seek to provide a system that provides ongoing interaction between learning of the user and training of the machine learning models that can be used to continuously optimise the stimuli available to all users and in particular to tune those used for particular users.

Preferably, there are four main aspects to the modelling, which are all interdependent and, preferably, occur simultaneously and some of which may optionally be repeated periodically or upon a change to the user interface device, apps installed etc:.

The creation of a model requires sufficient data about an individual's capabilities and preferences. In order to acquire this information, the user will initially go through an onboarding process aimed at testing aspects of communication protocols to determine the abilities and preferences of that individual. This means that the initial learning process is optimised and the stimulation protocols and language structure is personalised from the onset.

The main factors extracted for the haptic profile may include:.

Aside from the experimental data, the haptic profile may also contain other information that could impact on the choice of stimulus protocols and structure of the language. It is important to note that while this information might be used to inform parameter selection, the main purpose is to complement the experimental data from the testing phase.

In order to optimise the experience, the stimulation parameters in the haptic profile will preferably be continually tweaked to optimise the efficiency of the stimulation. This may be done in several ways:.

This aspect involves assigning stimulation protocols (or sets thereof) to applications through an app-store. This may involve the addition of new applications or an expansion of the functionality of an existing application.

The private and personalised nature of the language protocols makes the user interface system uniquely portable, which means it can be used across multiple haptic devices while maintaining consistent and effective communication. i.e. once a user has learned the association between a stimulation protocol and a meaning on one device, this same protocol could be employed to transmit the same information on multiple devices.

It will be appreciated that the data repository may take various forms including a central or distributed file store, database (such as SQL or other relational or non-relational database types). It may be implemented using storage devices such as hard disks, random access memories, solid state disks or any other forms of storage media. It will also be appreciated that the controller discussed herein may represent a single processor or a collection of processors acting in a synchronised, semi-synchronised or asynchronous manner. For example, a master server system may maintain the user interface system and be responsible for interaction between fully autonomous agents whilst a user's local PC or other computing device may be the "controller" or I/O device responsible for communicating with the user interface device. Alternatively, a central system may be responsible for processing decisions. The machine learning system may take many forms including a trained neural network, Support Vector Machines, Random Forest, K-Means, Matrix Factorisation or some combination thereof.

The method and system enables stimuli to be harmonised across devices for a user and in the case where different stimulators are used, the system and method could approximate a stimulus pattern so that it is as close as possible depending on the capabilities of the different stimulators and recognised to be the same by the user across those different devices.

The above discussion sets out principles and details of the underlying system and method that can be applied to different forms of user interface that may use differing (or multiple) types of stimuli and/or stimulators. It will furthermore be appreciated that it also opens up opportunities for more complex stimulators to be developed without the need to identify exactly which stimulus patterns/types they will use as these can be tailored to the user during deployment or later.

An example of this approach is discussed below with reference to <FIG>. In this arrangement, a haptic stimulator is used that is controllable such that one or more of frequency, amplitude, duration, signal gaps, waveform shape or other attributes can be adjusted to suit the user. The Figures illustrate:.

<FIG> is a schematic diagram showing clustering users on the basis of the similarity of their haptic profile. Stimulus patterns are also clustered based on their similarity and emotional context. This is used by the system to match stimulus patterns to both appropriate haptic profiles and to appropriate applications.

The system may be configured to automatically or semi-automatically assign stimuli to functions/applications of a device. The content of the stimulus (such as haptic) profile and the type of content/function may be matched to provide either an automatic assignment of or a narrowed down selection of "appropriate" stimuli and not the full complement of stimuli in the profile.

In addition to physical reactions to stimuli, emotional reaction may also be tested/recorded. For example, an emotional valence value may be assessed either by monitoring reactions, directly asking users about testing and/or using prediction based on other data from the user, e.g. performance/perception data from various psychological tests. Physiological measures e.g. heart rate, temperature and movement may also be used to assess the emotional response to a stimulus.

Stimuli may be grouped, either by emotion (happy, enjoyable, scary. ), activity type (pleasure, business, family), for contacts (a group for spouse etc) or in other ways such that a stimulus from that group is assigned to alerts or actions from the device associated with that grouping. Preferably, stimuli are classified according to what effect they will have on you, whether it be emotional associations, e.g. long duration pulses, low frequency and slow modulations for calming stimuli or short duration pulses in a higher frequency-band with fast modulations. for some notifications/alerts.

One possible arrangement may include the steps:.

It is to be appreciated that certain embodiments of the invention as discussed below may be incorporated as code (e.g., a software algorithm or program) residing in firmware and/or on computer useable medium having control logic for enabling execution on a computer system having a computer processor. Such a computer system typically includes memory storage configured to provide output from execution of the code which configures a processor in accordance with the execution. The code can be arranged as firmware or software, and can be organized as a set of modules such as discrete code modules, function calls, procedure calls or objects in an object-oriented programming environment. If implemented using modules, the code can comprise a single module or a plurality of modules that operate in cooperation with one another.

Claim 1:
A user interface system (<NUM>) comprising a controller (<NUM>), a data repository (<NUM>) and an I/O unit (<NUM>), the user interface system being configured to cause, via the I/O unit, a user interface device (<NUM>) of a user to provide a stimulus to the user, the user interface system is further configured to receive, from the user, data on the stimulus, wherein in response to the stimulus and the data provided by the user on the stimulus, the user interface system is arranged to record, in the data repository, stimulus recognised by the user,
the controller being operable to determine an optimised set of stimuli to be used by the user interface device in dependence on the stimuli recognised by the user, the optimised set matching the number of stimuli provided by the user interface device,
the controller being arranged to communicate with the user interface device and configure the user interface device to use the optimised set of stimuli for its functions, characterised in that the system is configured to communicate with multiple user devices of a user and is configured to provide a substantially consistent stimulus for a particular function across the multiple user devices.