Patent Description:
Modern cars and other vehicles are frequently provided with touchscreens which can be used for controlling at least part of a car's functionality, such as heater settings, audio playout, route planning, and the like. Whereas some manufacturers mix touchscreen-based user interfaces (Uls) and conventional, physical buttons, sliders, levers, etc, an increasing use of touchscreen-based Uls in vehicles is expected.

A problem which arises with the emerging use of touchscreen-based Uls in vehicles is the challenge for the driver to simultaneously maintain attention to the traffic and safely maneuver the car, while operating the touchscreen. This is aggravated by the fact that modern cars are provided with more and more functionality, leading to increasingly complex Uls with many virtual buttons, knobs, sliders, and other UI elements. In addition, in contrast to physical buttons, the spatial arrangement of the UI elements displayed on the touchscreen, i.e., the virtual buttons, sliders, etc, is dynamic and changes depending on context, which makes it more difficult for the driver to locate the UI element which she/he intends to touch.

Gesture control has been offered as a means of touchless interaction but is typically limited to a set of few gestures (see, e.g., https://www. digitaltrends. com/cars/<NUM>-bmw-<NUM>-series-gesture-control-pictures-video-news/, retrieved on <NUM> March <NUM>).

Haptic technology can be used for providing the sensation of touch when interacting with a touchscreen, in an attempt to mimic the sensation when touching a physical button. Haptic feedback can create an experience of touch by applying forces, vibrations, or motions, to the object touching the touchscreen, such as the finger of the driver. Thereby, finding the correct button on an in-vehicle touchscreen, without looking at the touchscreen while driving, is simplified (see, e.g., https://www. bosch-mobility-solutions. com/en/solutions/infotainment/display-and-interaction-systems/, retrieved on <NUM> March <NUM>, https://www. com/enterprise/automotive/, retrieved on <NUM> March <NUM>).

However, due to the steadily increasing complexity of in-vehicle touchscreen-based Uls, finding the intended button or other UI element even with haptic feedback is a challenge, as it attracts the driver's attention and prevents the driver from keeping both hands at the steering wheel while operating the touchscreen.

<CIT> discloses solutions to provide haptic sensations as a function of eye gaze, e.g., a system which includes a detector configured to determine a direction of an eye gaze of a user of the system, a processor configured to generate a signal representative of a haptic effect based on the direction of the eye gaze, and a haptic output device configured to receive the signal from the processor and output the haptic effect to the user.

<CIT> discloses an eye-tracking device configured to track a user's eye gaze while looking at media content comprising an image, and a processor configured to execute one or more computer program modules, including a content determination module that, when executed by the processor, analyzes the media content to identify the image, an eye gaze determination module that, when executed by the processor, determines a gaze location of the user's eye gaze while looking at the image, and an event determination module that, when executed by the processor, determines an event to trigger based on the identification of the image and the gaze location.

It is an object of the invention to provide an improved alternative to the above techniques and prior art.

More specifically, it is an object of the invention to provide improved touchscreen-based Uls. In particular, it is an object of the invention to provide improved touchscreen-based Uls which diminish the time the user is required to look at the touchscreen to locate a UI element which the user intends to touch.

These and other objects of the invention are achieved by means of different aspects of the invention, as defined by the independent claims. Embodiments of the invention are characterized by the dependent claims.

According to a first aspect of the invention, a computing device for controlling a haptic touchscreen is provided. The computing device comprises processing circuitry which causes the computing device to become operative to display a plurality of UI elements on the touchscreen. The computing devices becomes further operative to acquire information pertaining to a point of gaze of a user gazing at the touchscreen. The information pertaining to the point of gaze is acquired from a gaze detector. The computing devices becomes further operative to select at least one of the displayed UI elements based on the point of gaze. The computing devices becomes further operative to control the touchscreen to render the selected UI elements haptically distinguishably from the other UI elements.

According to a second aspect of the invention, a method of controlling a haptic touchscreen is provided. The method is performed by a computing device and comprises displaying a plurality of UI elements on the touchscreen. The method further comprises acquiring information pertaining to a point of gaze of a user gazing at the touchscreen. The information pertaining to the point of gaze is acquired from a gaze detector. The method further comprises selecting at least one of the displayed UI elements based on the point of gaze. The method further comprises controlling the touchscreen to render the selected UI elements haptically distinguishably from the other UI elements.

According to a third aspect of the invention, a computer program is provided. The computer program comprises instructions which, when the computer program is executed by a computing device, causes the computing device to carry out the method according to an embodiment of the second aspect of the invention.

According to a fourth aspect of the invention, a computer-readable data carrier is provided. The data carrier has stored thereon the computer program according to an embodiment of the third aspect of the invention.

According to a fifth aspect of the invention, a data carrier signal is provided. The data carrier signal carries the computer program according to an embodiment of the third aspect of the invention.

In the present context, "haptically distinguishable" means that the selected UI elements on the one hand, and the other (not selected) UI elements on the other hand, are rendered with a haptic contrast relative to each other, i.e., with respective haptic properties which are sufficiently different such that the user touching the UI elements (e.g., using his/her finger) can sense a difference between the selected UI elements and the other UI elements.

The invention makes use of an understanding that UI elements which the user is gazing at can be selected to be "haptically highlighted" to facilitate selecting an intended UI element by the user, i.e., a UI element which the user intends to touch or actuate, without gazing at the touchscreen for an extended duration of time. In particular for in-vehicle touchscreens, reducing the time during which the driver gazes at the touchscreen, rather than watching the traffic around the vehicle, is advantageous since the safety of the driver, the vehicle, and its surroundings, are improved.

Even though advantages of the invention have in some cases been described with reference to embodiments of the first aspect of the invention, corresponding reasoning applies to embodiments of other aspects of the invention.

Further objectives of, features of, and advantages with, the invention will become apparent when studying the following detailed disclosure, the drawings and the appended claims. Those skilled in the art realize that different features of the invention can be combined to create embodiments other than those described in the following.

The above, as well as additional objects, features and advantages of the invention, will be better understood through the following illustrative and non-limiting detailed description of embodiments of the invention, with reference to the appended drawings, in which:.

The invention will now be described more fully herein after with reference to the accompanying drawings, in which certain embodiments of the invention are shown. Rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

<FIG> illustrates a driver <NUM> of a car <NUM> operating an in-vehicle touchscreen <NUM>, in accordance with embodiments of the invention. The touchscreen <NUM> may, e.g., provide the UI of a car infotainment system. The touchscreen <NUM> is a haptic touchscreen and is controlled by a computing device <NUM> for controlling the haptic touchscreen <NUM>, which is schematically illustrated in <FIG> and described in more detail below.

The haptic touchscreen <NUM> may be a touchscreen with integrated haptic capabilities. For instance, the haptic touchscreen may be based on Electroactive Polymers (EAPs), which are deposited in a multilayer structure on top of a (non-haptic) touchscreen to provide spatially resolved actuation by application of an electric field (see, e.g., <CIT>, <CIT>). Thereby, a change in texture, vibrations, forces, or motion, can be rendered for sensing by an object interacting with, i.e., touching, the touchscreen <NUM>, such as the finger <NUM> of the driver <NUM>. The haptic touchscreen <NUM> may alternatively be based on other known technologies for haptic screen actuation, such as piezoelectric actuators, shape-shifting materials, linear resonant actuators, UV shape polymers, etc..

The haptic touchscreen <NUM> may alternatively comprise a conventional, non-haptic touchscreen, i.e., a touch-sensitive display, in combination with a haptic actuator which may be provided separately from the (non-haptic) touchscreen. For instance, the haptic actuator may be based on ultrasonic haptic technology which enables creating a haptic sensation mid-air. Ultrasonic haptic technology utilizes ultrasonic focusing technology and modulation to apply desired tactile sensory stimuli to a certain point in mid-air, by controlling the phase and intensity of ultrasound pulses which are emitted by a set of ultrasound transducers. Such ultrasonic haptic actuator may be provided adjacent to the touchscreen <NUM>, e.g., on top of the touchscreen <NUM> so as to provide a haptic sensation to the finger <NUM> of the driver <NUM> when approaching, being close to, or touching, the touchscreen <NUM>. As an example, Ultraleap offers a mid-air haptics UI for automotive applications (https://www. com/enterprise/automotive/, retrieved on retrieved on <NUM> March <NUM>).

With reference to <FIG>, the computing device <NUM> for controlling a haptic touchscreen <NUM> (in the following referred to as "touchscreen") comprises processing circuitry <NUM>. The processing circuitry <NUM> may comprise one or more processors <NUM>, such as Central Processing Units (CPUs), microprocessors, application processors, application-specific processors, Graphics Processing Units (GPUs), and Digital Signal Processors (DSPs) including image processors, or a combination thereof, and a memory <NUM> comprising a computer program <NUM> comprising instructions. When executed by the processor(s) <NUM>, the instructions cause the computing device <NUM> to become operative in accordance with embodiments of the invention described herein. The memory <NUM> may, e.g., be a Random-Access Memory (RAM), a Read-Only Memory (ROM), a Flash memory, or the like. The computer program <NUM> may be downloaded to the memory <NUM> by means of a network interface circuitry (not shown in <FIG>), as a data carrier signal carrying the computer program <NUM>. The network interface circuitry may comprise one or more of a cellular modem (e.g., GSM, UMTS, LTE, <NUM>, or higher generation), a WLAN/Wi-Fi modem, a Bluetooth modem, an Ethernet interface, an optical interface, or the like, for exchanging data between the computing device <NUM> and other computing devices, communications devices, a radio-access network, and/or the Internet. The processing circuitry <NUM> may alternatively or additionally comprise one or more Application-Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or the like, which are operative to cause the computing device <NUM> to become operative in accordance with embodiments of the invention described herein.

The computing device <NUM> may further comprise one or more interface circuitries <NUM> ("I/O" in <FIG>) for communicating with the haptic touchscreen <NUM> and the gaze detector <NUM>, and optionally other external circuits or devices. The one or more interface circuitries <NUM> may be based on Universal Serial Bus (USB), the Vehicle Communication Interface (VCI), or the like.

More specifically, the processing circuitry <NUM> causes the computing device <NUM> to become operative to display a plurality of UI elements, aka virtual UI elements, on the touchscreen <NUM>. In particular, this may be UI elements which the driver <NUM> (or a user of the computing device <NUM> in general) can interact with using a finger <NUM>, a stylus pen, or a similar object, by means of touching, pressing, clicking, sliding, dragging, or the like. Typically, such UI elements can be used for controlling an operation or operations of the computing device <NUM> or another device or apparatus which is operatively connected to, and controlled by, the computing device <NUM>. The operation or operations may relate to settings or configuration of a device or an apparatus, such as a smartphone, a tablet, a computer, a household appliance, or a vehicle <NUM> such as a car. For example, the UI elements may be used for controlling a media playout device such as a radio, television, or music player, a heater or air condition, lights, etc. Each UI element may, e.g., be any one of a button, a knob, a dial, a slider, a toggle switch, a wheel, or the like.

The computing device <NUM> is further operative to acquire information pertaining to a point of gaze <NUM> of a user, e.g., the driver <NUM>, gazing at the touchscreen <NUM>. The point of gaze <NUM> is the location on the touchscreen <NUM> where the direction of gaze <NUM> of the driver <NUM> intersects the surface of touchscreen <NUM>. The information pertaining to the point of gaze may be the point of gaze <NUM>, i.e., where the driver <NUM> is gazing on the touchscreen <NUM> (i.e., a position), expressed relative to the display area of the touchscreen <NUM>, e.g., in terms of a coordinate system associated with the display area. Alternatively, the information pertaining to the point of gaze may be a position of the driver's <NUM> eye(s) or head relative to the touchscreen <NUM>, and a direction <NUM> of the driver's <NUM> gaze. Based on this information, the point of gaze <NUM> can be calculated as the intersection of the driver's direction <NUM> of gaze with the surface of the touchscreen <NUM>. As yet a further alternative, the information may be an identification of a displayed UI element at which the driver <NUM> is gazing.

The information pertaining to the point of gaze is acquired from a gaze detector <NUM>, aka eye tracker. The gaze detector <NUM> may be comprised in the computing device <NUM> or provided separate from, and operatively connected to, the computing device <NUM>. For instance, the gaze detector <NUM> may comprise a light source and a camera. Gaze detection, aka eye tracking, is well known in the art and can be performed with below-centimeter accuracy. While different gaze detection technologies are known, the most widely used designs rely on light, typically infrared light, which is reflected from the eye and sensed by a digital camera, e.g., a video camera, or some other suitable optical sensor with spatial resolution. The captured information is then analyzed to extract eye rotation from changes in reflections. Such gaze detectors typically use the corneal reflection (the first Purkinje image) and the center of the pupil as features to track over time. A more sensitive type of gaze detector uses reflections from the front of the cornea (first Purkinje image) and the back of the lens (fourth Purkinje image) as features to track. A still more sensitive method of gaze detection is to image features from inside the eye, such as the retinal blood vessels, and follow these features as the eye rotates. Optical methods are widely used for gaze detection and are favored for being non-invasive and inexpensive. Information acquired from the gaze detector <NUM> may also be used for determining whether the gaze <NUM> of the driver <NUM> is stable, a fixation duration (the time during which the eye or eyes rest(s) on an object in the surroundings, such as a displayed UI element), and/or a time the driver <NUM> gazes at the touchscreen <NUM>.

The computing device <NUM> is further operative to select at least one of the displayed UI elements based on the point of gaze <NUM>, as is described in further detail below, and to control the touchscreen <NUM> to render the selected UI elements haptically distinguishably from the other UI elements, i.e., the UI elements which have not been selected. The selected UI elements are also rendered haptically distinguishably from the remaining display area of the touchscreen <NUM> which is not occupied by displayed UI elements (i.e., areas in-between the UI elements). The haptic rendering of the selected UI elements is preferably aligned with the visual rendering, i.e., the displaying of the selected UI elements by the touchscreen <NUM>.

In the present context, "haptically distinguishable" means that the selected UI elements on the one hand, and the other (not selected) UI elements on the other hand, are rendered with a haptic contrast relative to each other, i.e., with respective haptic properties which are sufficiently different such that a user, e.g., the driver <NUM>, touching the UI elements (e.g., using his/her finger <NUM>) can sense a difference between the selected UI elements and the other UI elements. Thereby, the selected UI elements are "haptically highlighted" to facilitate selecting an intended UI element by the driver <NUM> without gazing at the touchscreen <NUM> for an extended duration of time. In other words, the selected UI elements are rendered so as to achieve a haptic contrast which makes it easier for the driver <NUM> to distinguish the UI elements which she/he likely intends to actuate from the other UI elements which she/he likely does not intend to actuate. The selection of the UI elements for haptically distinguishable rendering is based on where on the touchscreen <NUM> the driver <NUM> is gazing. In other words, the point of gaze <NUM> of the driver <NUM> on the touchscreen <NUM> is used as an indication of where on the touchscreen <NUM> the driver intends to touch with her/his finger <NUM>. By reducing the time during which the driver <NUM> gazes at the touchscreen <NUM>, rather than watching the traffic around the car <NUM>, safety of the driver <NUM>, the car <NUM>, and the surroundings are improved.

The computing device <NUM> may be operative to control the touchscreen <NUM> to render the selected UI elements haptically distinguishably from the other UI elements by rendering the selected UI elements with haptic properties which are different from haptic properties used for rendering the other UI elements. That is, all displayed UI elements are haptically rendered, but with different properties such that the selected UI elements can be distinguished by relying on the user's (e.g., the driver's <NUM>) sense of touch. As an alternative, the computing device <NUM> may be operative to control the touchscreen <NUM> to render the selected UI elements haptically distinguishably from the other UI elements by haptically rendering the selected UI elements but not haptically rendering the other UI elements. That is, only the selected UI elements are haptically rendered.

Rendering the selected UI elements haptically distinguishably from the other UI elements may, e.g., be achieved by controlling the touchscreen <NUM>, in particular the surface of the touchscreen <NUM>, to exhibit a difference in topology between the selected UI elements and the other UI elements, as well as the remaining surface area of the touchscreen <NUM> which is not occupied by UI elements. For instance, a difference in topology may be achieved by protruding the selected UI elements, or by creating groves at the positions of the selected UI elements, using EAPs which are deposited in a multilayer structure on top of the touchscreen <NUM> in a matrix structure to enable spatially-resolved actuation by application of an electric field (see, e.g., <CIT>, <CIT>). Alternatively, the haptic touchscreen <NUM> may be controlled to render the selected UI elements haptically distinguishably from the other UI elements by causing a change in texture, a force, a friction, or a vibration, which can be sensed by the finger <NUM>, or other object, when touching one of the selected UI elements on the touchscreen <NUM>. The other (non-selected) UI elements may either not be rendered haptically at all, or may be rendered with a change in texture, a force, a friction, or a vibration, which, when touching one of the other UI elements with the finger <NUM>, or other object, can be distinguished from the selected UI elements.

The computing device <NUM> may optionally further be operative to initiate an action associated with the UI element actuated by the user, e.g., the driver <NUM>. Examples of actions associated with a UI element include, but are not limited to, controlling the operation of computing device <NUM> or a device or apparatus controlled by the computing device <NUM>, and starting, stopping, or modifying, functionality performed by the computing device <NUM> or a device or apparatus controlled by the computing device <NUM>, such as increasing or decreasing playout volume of a music player, switching a light on or off, increasing or decreasing a temperature, etc..

In the following, and with reference to <FIG>, examples of alternatives for selecting at least one of the displayed UI elements based on the point of gaze <NUM>, for subsequent rendering in a haptically distinguishable manner to facilitate the correct selection by the user (e.g., the driver <NUM>) of a displayed UI element which the user intends to actuate, are described. <FIG> illustrate an arrangement of example UI elements <NUM>-<NUM> which are displayed on the touchscreen <NUM>: a numerical keypad <NUM>, a set <NUM> of buttons for selecting a radio station, a slider <NUM> for adjusting a front-rear balance of audio playout, and a set <NUM> of up-down buttons for adjusting the volume of audio playout. It will be appreciated that the UI elements <NUM>-<NUM> illustrated in <FIG> are for illustrative purposes and only exemplify possible types of UI elements, their arrangement on the touchscreen <NUM>, and functionality which is associated with the UI elements <NUM>-<NUM>. The point of gaze <NUM> is in <FIG> marked with a cross, and the selected UI elements are indicated with bold lines.

With reference to <FIG>, the computing device <NUM> may be operative to select at least one of the displayed UI elements <NUM>-<NUM> based on respective distances of the displayed UI elements <NUM>-<NUM> from the point of gaze <NUM>. For instance, one or more of the displayed UI elements <NUM>-<NUM> which are closest to, i.e., have the shortest distance to, the point of gaze <NUM> may be selected. In practice, the closest UI elements <NUM>-<NUM> may be selected based on a threshold distance, which in <FIG> is indicated as a dashed circle <NUM> (the threshold distance is the radius of the circle <NUM>). As illustrated in <FIG>, all displayed UI elements <NUM>-<NUM> which have at least part of their area within the threshold distance from the point of gaze <NUM>, as marked by the circle <NUM>, are selected (in the example in <FIG>, buttons <NUM> "<NUM>" and "<NUM>", button <NUM> "P1", the slider <NUM>, and the up-down buttons <NUM> are selected). Optionally, if the computing device <NUM> is operative to select displayed UI elements <NUM>-<NUM> within a threshold distance (marked by the circle <NUM>) from the point of gaze <NUM>, the threshold distance may decrease with an increasing fixation duration of the user's (e.g., the driver's <NUM>) gaze at the touchscreen <NUM>. This is based on the understanding that the longer the driver <NUM> gazes at the touchscreen <NUM>, and preferably gazes at substantially the same point of gaze <NUM>, the more likely it is that the point of gaze <NUM> reflects the position at which the driver intends to touch the touchscreen <NUM>. Information pertaining to the fixation duration may be acquired from the gaze detector <NUM>. In practice, the computing device <NUM> may be operative to select, and haptically render, displayed UI elements <NUM>-<NUM> based on an initial threshold distance which is gradually decreased with increasing fixation duration. Optionally, the threshold distance may be kept constant for a preset duration of time and gradually decreased after the preset duration of time (e.g., one second) has lapsed.

As an alternative, a predetermined number of the displayed UI elements <NUM>-<NUM> which are closest to the point of gaze <NUM> may be selected, e.g., the three closest UI elements <NUM>-<NUM>. This may be achieved by calculating respective distances between the displayed UI elements <NUM>-<NUM> (e.g., from their respective geometric center or the closest point of their circumference) and the point of gaze <NUM>, sorting the calculated distances in increasing order, and selecting the predetermined number of closest UI elements <NUM>-<NUM>, i.e., the predetermined number of UI elements <NUM>-<NUM> having the shortest distance to the point of gaze <NUM>.

With reference to <FIG>, the computing device <NUM> may be operative to select at least one of the displayed UI elements <NUM>-<NUM> based on the selected at least one UI element being displayed within a partial area of the touchscreen <NUM>, wherein the partial area encompasses the point of gaze <NUM>. In the present context, a partial area may, e.g., be the left, right, upper, or lower, half of the touchscreen <NUM>. In <FIG>, the vertical and horizontal bisections of the touchscreen <NUM> into left and right halves, as well as upper and lower halves, respectively, are indicated by dashed lines <NUM> and <NUM>. As a further example, a partial area of the touchscreen <NUM> may be one of the quadrants defined by the bisections <NUM> and <NUM>. In the example illustrated in <FIG>, the point of gaze <NUM> (marked by a cross, the reference number is omitted for the sake of clarity) is in the upper-right quadrant of the touchscreen <NUM>, and the displayed UI elements <NUM> and <NUM> are accordingly selected. As an alternative, a partial area of the touchscreen <NUM> may be an area, such as a two-dimensional geometric shape, with the point of gaze <NUM> at its center. The two-dimensional geometric shape may, e.g., be a circle (as illustrated in <FIG>), an ellipse, a square, a rectangle, or the like. Optionally, the computing device <NUM> may be operative to decrease a size of the partial area of the touchscreen <NUM> with an increasing fixation duration of the user's (e.g., the driver's <NUM>) gaze at the touchscreen <NUM>. The longer the driver <NUM> stares at the touchscreen <NUM>, and preferably at substantially the same point of gaze <NUM>, the more likely it is that the point of gaze <NUM> reflects the position at which the driver <NUM> intends to touch the touchscreen <NUM>. Accordingly, the number of displayed UI elements which need to be selected for haptically distinguishable rendering can be reduced.

With reference to <FIG>, the computing device <NUM> may be further operative to select at least one additional of the displayed UI elements <NUM>-<NUM> based on a spatial arrangement of the displayed UI elements <NUM>-<NUM>. For instance, the computing device <NUM> may be operative to select at least one additional of the displayed UI elements <NUM>-<NUM> based on the selected at least one UI element and the at least one additional UI element being arranged as a group of UI elements. This is exemplified in <FIG>, which illustrates the point of gaze <NUM> (marked as a cross, the reference number is omitted for the sake of clarity) coinciding with the button <NUM> "<NUM>". In this case, all buttons <NUM> which are arranged as a group, in this case the entire numerical keypad <NUM>, are selected in addition to the button <NUM> "<NUM>" which is selected based on the point of gaze <NUM>. Alternatively, if at least one of the displayed UI elements <NUM>-<NUM> is selected based on respective distances of the displayed UI elements from the point of gaze, e.g., all displayed UI elements <NUM>-<NUM> within a threshold distance (marked by the dashed circle in <FIG>, the reference numeral is omitted for the sake of clarity) from the point of gaze <NUM>, the entire numerical keypad <NUM> is selected in addition to the buttons <NUM> "<NUM>", "<NUM>", "<NUM>", "*", "<NUM>", and "#", which are selected based on the threshold distance (marked by the dashed circle) from the point of gaze <NUM>.

The computing device <NUM> may optionally be further operative to select at least one additional of the displayed UI elements <NUM>-<NUM> based on related functionality which is controlled by the selected at least one UI element and the at least one additional UI element. In other words, displayed UI elements <NUM>-<NUM> which are related in terms of the functionality, operations, or actions, which they control, are selected. For instance, and with reference to <FIG>, if one of more buttons which are part of the numerical keypad <NUM> are selected, either based on the point of gaze <NUM> or respective distances of the UI elements from the point of gaze <NUM>, all remaining buttons which belong to the numerical keypad <NUM> are selected. As a further example, if one of the buttons <NUM> for selecting a preset radio station would be selected (e.g., "P1"), either based on the point of gaze <NUM> or respective distances of the buttons <NUM> for selecting a preset radio station from the point of gaze <NUM>, the other buttons <NUM> for selecting a preset radio station (e.g., "P2" and "P3") would be additionally selected.

With reference to <FIG>, the computing device <NUM> may be further operative to select at least one additional of the displayed UI elements <NUM>-<NUM> based on an actuation of a displayed UI element <NUM>-<NUM> by the user, e.g., the driver <NUM>. In the present context, an actuation of a displayed UI element means that the driver is interacting with, i.e., touching using a finger <NUM>, a stylus pen, or a similar object, a displayed UI element so as to initiate an action, operation, or functionality, which is associated with the UI element. In other words, the driver is clicking on the UI element. The at least one additional of the displayed UI elements <NUM>-<NUM> which is selected may, e.g., be a UI element which typically is selected subsequent to the actuated UI element. For instance, the at least one additional UI element may be related to the actuated UI element in terms of functionality. For the example illustrated in <FIG>, the driver <NUM> has actuated the button <NUM> "P1" for selecting a radio station. In response to detecting that the button <NUM> "P1" has been actuated, the UI elements <NUM> (e.g., for adjusting a front-rear balance of audio playout) and <NUM> (e.g., for adjusting the volume of audio playout) are selected as additional UI elements for haptically distinguishable rendering because the driver <NUM> is likely to adjust audio-playout settings after switching on the radio or switching to a different radio station. The at least one additional of the displayed UI elements <NUM>-<NUM> may, e.g., be selected based on stored associations between, or sequences of, UI elements. These associations or sequences may either be configured by a provider of the computing device, or a software application which is executed on the computing device <NUM>, or by the driver <NUM>. As an alternative, the computing device <NUM> may be operative to learn such associations or sequences during operation, by recording and statistically evaluating which of the displayed UI elements <NUM>-<NUM> are more likely to be sequentially actuated by the driver <NUM> than others.

The computing device <NUM> may optionally be operative to select at least one of the displayed UI elements <NUM>-<NUM> further based on context information. Context information may, e.g., relate to weather, ambient noise, road conditions, traffic conditions, a current configuration of the vehicle <NUM>, a number of passengers in the vehicle <NUM>, or the like. As an example, if a passenger seat is not occupied, the driver <NUM> is unlikely to activate a heat heater for the unoccupied seat, and the corresponding UI element is consequently not selected for haptically distinguishable rendering. On the contrary, UI elements for controlling heat seaters of occupied passenger seats are selected for haptically distinguishable rendering. As a further example, if the environment is relatively bright, the driver <NUM> is unlikely to switch on the car's <NUM> lights, and the corresponding UI element is consequently not selected for haptically distinguishable rendering. On the contrary, UI elements for activating the car's <NUM> lights are selected for haptically distinguishable rendering if the ambient light conditions mandate using lights.

The computing device <NUM> may optionally be operative to adjust one or more of a size of the selected UI element(s), and a haptic contrast between the selected UI element(s) and the other UE elements, based on a duration of time the user, e.g., the driver <NUM>, gazes at the touchscreen <NUM>. Preferably, the size of the UI elements is adjusted based on the fixation duration, i.e., the time duration of time during which the point of gaze <NUM> is substantially stable. For example, the selected UI elements may be haptically rendered with an increased size, i.e., area of the UI elements, if the driver <NUM> gazes at the touchscreen <NUM> only during a short time interval, e.g., a few tens of a second. This is advantageous in that the acquired point of gaze <NUM> may not reliably reflect the position <NUM> on the touchscreen <NUM> which the driver <NUM> intends to touch, as compared to situations when the driver <NUM> gazes at the touchscreen <NUM> during a longer time interval, e.g., one or two seconds. Thereby, the diver's <NUM> actuation of the intended UI element is facilitated. As an alternative, or in addition, to adjusting a size of the selected UI element(s), a haptic contrast between the selected UI element(s) and the other UE elements may be adjusted so as to facilitate identifying the intended UI element by the driver <NUM>.

The computing device <NUM> may optionally be further operative to re-select at least one of the displayed UI elements <NUM>-<NUM> based on a change in the point of gaze <NUM>. For instance, this may be the case if the driver <NUM> gazes at the touchscreen <NUM> for a while, but subsequently gazes at a different point of the touchscreen <NUM>, either directly (without stop gazing at the touchscreen <NUM>) or interrupted by gazing in a different direction (e.g., at the street in front of the car <NUM>) during a short duration of time. In this case, the selection of at least one of the displayed UI elements <NUM>-<NUM> based on the previous point of gaze may be reset, and at least one of the displayed UI elements <NUM>-<NUM> is selected based on the current point of gaze <NUM>. Preferably, re-selection of at least one of the displayed UI elements <NUM>-<NUM> is only performed if the change in the point of gaze <NUM> exceeds a threshold distance, e.g., a few millimeters or a centimeter.

The computing device <NUM> may optionally be operative to select at least one of the displayed UI elements <NUM>-<NUM>, and/or to control the touchscreen <NUM> to render the selected UI elements haptically distinguishably from the other UI elements, in response to one or more of: detecting that the user (e.g., the driver <NUM>) is gazing at the touchscreen <NUM>, detecting a touch input to the touchscreen <NUM>, detecting that an object (e.g., the finger <NUM> or a stylus pen) is approaching the touchscreen <NUM>, and determining that the point of gaze <NUM> is substantially stable (i.e., the gaze <NUM> of the driver <NUM> is substantially stable). Detecting a touch input to the touchscreen <NUM> may, e.g., be based on information received from the touchscreen <NUM> that a touch input to the touchscreen <NUM> has been detected, by means of the finger <NUM>, a stylus pen, or other object, irrespective of a position <NUM> indicating where on the touchscreen <NUM> the touch input was received. Detecting that an object, such as the finger <NUM>, is approaching the touchscreen <NUM> may, e.g., be based on information from the gaze detector <NUM>. This may be the case if the gaze detector <NUM> comprises a digital camera capable of identifying an object which is approaching the touchscreen <NUM>. Alternatively, an object approaching the touchscreen <NUM> may be detected by utilizing a LiDAR or a capacitive touchscreen which is capable of detecting that the finger <NUM> is hovering over the capacitive surface of the touchscreen <NUM>.

In particular, the computing device <NUM> may be operative to select at least one of the displayed UI elements <NUM>-<NUM>, and/or to control the touchscreen <NUM> to render the selected UI elements haptically distinguishably from the other UI elements in response to detecting that the user (e.g., the driver <NUM>) is gazing at the touchscreen <NUM>, detecting that an object (such as the finger <NUM> or a stylus pen) is approaching the touchscreen <NUM> (e.g., using the gaze detector <NUM> or capacitive touchscreen, as described hereinbefore, a separate camera, or a LiDAR), and detecting that the driver ceases gazing at the touchscreen <NUM>. This sequence of events is indicative of a typical operation of the touchscreen <NUM> by the driver <NUM>. More specifically, the driver <NUM> starts gazing at the touchscreen <NUM> to identify the intended UI element or elements, i.e., the UI element(s) which she/or intends to touch or actuate. Then, the driver starts extending her/his finger <NUM> towards the touchscreen <NUM>, while still gazing at the touchscreen <NUM>. This is likely initiated by the driver's <NUM> hand releasing the wheel and moving towards the touchscreen <NUM>. Once the driver <NUM> has started extending her/his finger <NUM> towards the touchscreen <NUM>, into the direction of the intended point of touch <NUM>, she/he can cease, i.e., stop or discontinue, gazing at the touchscreen <NUM> to focus her/is attention back to the road or the traffic.

The computing device <NUM> may optionally be operative to control the touchscreen <NUM> to cease, i.e., stop or discontinue, rendering the selected UI elements haptically distinguishably from the other UI elements in response to detecting that a predetermined duration of time since the user (e.g., the driver <NUM>) has started gazing at the touchscreen <NUM> has expired without detecting a touch input to the touchscreen <NUM>. This may, e.g., be the case if the driver has accidentally gazed at the touchscreen <NUM>, without intending to touch the touchscreen <NUM>, or has changed her/his mind after gazing at the touchscreen <NUM>. Even further, the driver <NUM> may have gazed at the touchscreen <NUM> to read information, e.g., an inside or outside temperature, or to check which radio station is currently selected, without the intention to touch the touchscreen <NUM> or a displayed UI element <NUM>-<NUM>.

The computing device <NUM> may optionally be further operative to correct the point of gaze <NUM> by a gaze-point offset which is determined based on preceding touch inputs received by the touchscreen <NUM>. For instance, the computing device <NUM> may be operative, in response to receiving touch inputs to the touchscreen <NUM>, to calculate an offset vector (distance and direction) between the position <NUM> of the touch input on the touchscreen <NUM> and the point of gaze <NUM> preceding the touch input. By averaging offset vectors over a number of touch inputs, an estimate is obtained which can be used for correcting a systematic offset between the point of gaze <NUM>, i.e., where the user (e.g., the driver <NUM>) is gazing prior to selecting a displayed UI element <NUM>-<NUM>, and the position <NUM> of a subsequent touch input by the driver, which reflects where on the touchscreen <NUM> the driver intended to touch. The calculated offset vectors may optionally be calculated and/or stored based on the point of gaze <NUM>, i.e., as a function of where on the touchscreen <NUM> the driver <NUM> had gazed before touching the touchscreen <NUM>. Thereby, any dependence of the systematic offset on where on the touchscreen <NUM> the driver <NUM> gazes is taken into consideration.

With reference to <FIG>, the computing device <NUM> may optionally be further operative to detect a first touch input to the touchscreen <NUM>, and to detect a second touch input to the touchscreen <NUM>, wherein the second touch input corresponds to an actuation of a displayed UI element <NUM> by the user, e.g., the driver <NUM>. The computing device <NUM> is further operative to, if the second touch input is detected within a threshold time interval from the first touch input, and a position <NUM> of the second touch input is within a threshold distance from a position <NUM> of the first touch input, control the touchscreen <NUM> to display the actuated UI element <NUM> at the first position. This embodiment applies to scenarios when the driver <NUM> has touched the touchscreen <NUM> at a first position <NUM>, but has missed the displayed UI element which she/he had intended to actuate, e.g., the arrow-up button <NUM> illustrated in <FIG>. When realizing that she/he has not found the intended UI element, e.g., because the first touch input occurred at a position <NUM> where no haptically highlighted UI element is displayed, the driver <NUM> performs a second touch input at a second position <NUM>, as a means to correct the first touch input. In order to discriminate intended subsequent touch inputs at the second position <NUM> from corrective subsequent touch inputs, only second touch inputs at a second position <NUM> which are within a threshold distance from the position <NUM> of the first touch input (e.g., one or a few centimeters), and which occur within a threshold time interval from the first touch input (e.g., one or a few seconds), are considered to represent corrective actions by the driver <NUM>. In response thereto, the actuated UI element <NUM> is displayed at the position <NUM> where the first touch input has occurred, because the driver <NUM> had expected the actuated UI element <NUM> to be displayed at the first position <NUM>. In practice, this may be achieved by moving the actuated UI element <NUM> such that is area encompasses the first position <NUM> (e.g., by re-arranging the displayed UI elements on the touchscreen <NUM>), or to increase a size of the actuated UI element <NUM> such that is area encompasses the first position <NUM>. Optionally, this corrective action may be applied only after the driver <NUM> has corrected a first touch input several times, with the vector difference between the position <NUM> of the second touch input and the position <NUM> of the first touch input being relatively constant, as this is an indication that the driver <NUM> missing the intended UI element by a margin is a systematic error in the driver's <NUM> estimation of where to touch the touchscreen <NUM> without gazing at the touchscreen <NUM>. Optionally, the computing device <NUM> is further operative to, if the first touch input triggers an action, detect that the user (e.g., the driver <NUM>) reverses the action triggered by the first touch input before the second touch input is received. For instance, this may be the case if the first touch input corresponds to an actuation of a displayed UI element, which is associated with an action such as lowering a playout volume or switching off music playout, by the driver. Such situation may occur if a displayed UI element is present at the position <NUM> of the first touch, but the displayed UI element is not the UI element which the driver <NUM> intended to touch or actuate. In such scenario, the driver <NUM> may first cancel the unintended action, e.g., by raising the volume or switching on music playout again, before attempting to trigger the intended action by the performing the second touch input at the second position <NUM>.

Even though embodiments of the invention are described in relation to a haptic touchscreen which is comprised in a vehicle, such as the haptic touchscreen <NUM> illustrated in in <FIG>, embodiments of the invention are not limited to in-vehicle haptic touchscreens. For instance, the computing device <NUM> for controlling a haptic touchscreen may alternatively be embodied as a smartphone, a tablet, or a computer screen, thereby enabling a user to control operation of the computing device <NUM>, or any other device or apparatus operatively connected to the computing device <NUM>, without looking at the touchscreen <NUM> for an extended duration of time. For instance, this may allow users to operate their smartphones without retrieving them from a pocket where they are stored. As a further example, a user of a smartphone or tablet may operate her/his computing device while watching a presenter without looking at the touchscreen of her/his smartphone or tablet for an extended duration of time.

In the following, embodiments of a method <NUM> of controlling a haptic touchscreen <NUM> are described with reference to <FIG>. The method <NUM> is performed by a computing device <NUM> and comprises displaying <NUM> a plurality of UI elements <NUM>-<NUM> on the touchscreen <NUM>. The method <NUM> further comprises acquiring <NUM> information pertaining to a point of gaze <NUM> of a user <NUM> gazing at the touchscreen <NUM>. The information pertaining to the point of gaze <NUM> is acquired from a gaze detector <NUM>. The method <NUM> further comprises selecting <NUM> at least one of the displayed UI elements <NUM>-<NUM> based on the point of gaze <NUM>. The method <NUM> further comprises controlling <NUM> the touchscreen <NUM> to render the selected UI elements haptically distinguishably from the other UI elements. For instance, the touchscreen <NUM> may be controlled to render the selected UI elements haptically distinguishably from the other UI elements by rendering the selected UI elements with haptic properties different from haptic properties used for rendering the other UI elements. Alternatively, the touchscreen <NUM> may be controlled to render the selected UI elements haptically distinguishably from the other UI elements by haptically rendering the selected UI elements but not haptically rendering the other UI elements.

The at least one of the displayed UI elements <NUM>-<NUM> may be selected <NUM> based on respective distances of the displayed UI elements <NUM>-<NUM> from the point of gaze <NUM>. Optionally, the selecting <NUM> at least one of the displayed UI elements <NUM>-<NUM> may comprise selecting displayed UI elements <NUM>-<NUM> within a threshold distance from the point of gaze <NUM>, wherein the threshold distance decreases with an increasing fixation duration of the user's gaze <NUM> at the touchscreen <NUM>.

The least one of the displayed UI elements <NUM>-<NUM> mat alternatively be selected <NUM> based on the selected at least one UI element being displayed within a partial area the touchscreen <NUM>, wherein the partial area encompasses the point of gaze <NUM>. Optionally, the method <NUM> may further comprise decreasing <NUM> a size of the partial area of the touchscreen <NUM> with an increasing fixation duration of the user's gaze <NUM> at the touchscreen <NUM>.

The method <NUM> may further comprise selecting <NUM> at least one additional of the displayed UI elements <NUM>-<NUM> based on a spatial arrangement of the displayed UI elements <NUM>-<NUM>. Optionally, the least one additional of the displayed UI elements <NUM>-<NUM> may be selected <NUM> based on the selected at least one UI element and the at least one additional UI element being arranged as a group of UI elements.

The method <NUM> may further comprise selecting <NUM> at least one additional of the displayed UI elements <NUM>-<NUM> based on related functionality which is controlled by the selected at least one UI element and the at least one additional UI element.

The method <NUM> may further comprise selecting <NUM> at least one additional of the displayed UI elements <NUM>-<NUM> based on an actuation of a displayed UI element <NUM>-<NUM> by the user <NUM>.

Optionally, the at least one of the displayed UI elements <NUM>-<NUM> may be selected further based on context information.

The method <NUM> may further comprise adjusting <NUM> one or more of: a size of the selected UI element(s), and a haptic contrast between the selected UI element(s) and the other UI elements, based on a duration of time the user <NUM> gazes at the touchscreen <NUM>.

The method <NUM> may further comprise re-selecting at least one of the displayed UI elements <NUM>-<NUM> based on a change in the point of gaze <NUM>.

The at least one of the displayed UI elements <NUM>-<NUM> is selected <NUM> and/or the touchscreen <NUM> is controlled <NUM> to render the selected UI elements haptically distinguishably from the other UI elements in response to one or more of <NUM>: detecting that the user <NUM> is gazing at the touchscreen <NUM>, detecting a touch input to the touchscreen <NUM>, detecting that an object <NUM> is approaching the touchscreen <NUM>, and determining that the point of gaze <NUM> is substantially stable.

The at least one of the displayed UI elements <NUM>-<NUM> is selected <NUM> and/or the touchscreen <NUM> is controlled <NUM> to render the selected UI elements haptically distinguishably from the other UI elements in response to: detecting that the user is gazing at the touchscreen, detecting that an object is approaching the touchscreen, and detecting that the user ceases gazing at the touchscreen.

The method <NUM> may further comprise ceasing <NUM> rendering the selected UI elements haptically distinguishably from the other UI elements in response to detecting that a predetermined duration of time since the user <NUM> has started gazing at the touchscreen <NUM> has expired without detecting a touch input to the touchscreen <NUM>.

The method <NUM> may further comprise correcting the point of gaze <NUM> by a gaze-point offset determined based on preceding touch inputs received by the touchscreen <NUM>.

The method <NUM> may further comprise detecting <NUM> a first touch input to the touchscreen <NUM> and detecting <NUM> a second touch input to the touchscreen <NUM>. The second touch input corresponding to an actuation of a displayed UI element <NUM> by the user <NUM>. The method <NUM> may further comprises controlling <NUM> the touchscreen <NUM> to display the actuated UI element at the first position <NUM> if the second touch input is detected within a threshold time interval from the first touch input, and a position <NUM> of the second touch input is within a threshold distance from a position <NUM> of the first touch input. Optionally, the first touch input triggers an action, and the method <NUM> further comprises detecting <NUM> that the user <NUM> reverses the action triggered by the first touch input before the second touch input is received.

It will be appreciated that the method <NUM> may comprise additional, alternative, or modified, steps in accordance with what is described throughout this disclosure. An embodiment of the method <NUM> may be implemented as the computer program <NUM> comprising instructions which, when the computer program <NUM> is executed by the computing device <NUM> cause the computing device <NUM> to carry out the method <NUM> and become operative in accordance with embodiments of the invention described herein. The computer program <NUM> may be stored in a computer-readable data carrier, such as the memory <NUM>. Alternatively, the computer program <NUM> may be carried by a data carrier signal, e.g., downloaded to the memory <NUM> via the network interface circuitry (not shown in <FIG>).

Claim 1:
A computing device (<NUM>) for controlling a haptic touchscreen (<NUM>), the computing device comprising processing circuitry (<NUM>) causing the computing device to become operative to:
display a plurality of user-interface, UI, elements (<NUM>-<NUM>) on the touchscreen (<NUM>),
acquire information pertaining to a point of gaze (<NUM>) of a user (<NUM>) gazing at the touchscreen from a gaze detector (<NUM>),
characterized in that the computing device becomes further operative to select at least one of the displayed UI elements (<NUM>-<NUM>) based on the point of gaze (<NUM>), and control the touchscreen (<NUM>) to render the selected UI elements haptically distinguishable from the other UI elements in response to a sequence of:
detecting that the user (<NUM>) is gazing at the touchscreen (<NUM>), detecting that an object (<NUM>) is approaching the touchscreen (<NUM>), and detecting that the user (<NUM>) ceases gazing at the touchscreen (<NUM>).