Patent Description:
Mixed reality (MR) and Virtual Reality (VR) are foreseen to become an integral technology in the networked society and potently disrupt the consumer electronics market. Several companies believe that there will be a potential next platform shift to MR and VR, which can give growth to $<NUM> billion by <NUM>. Of this, MR is the bigger opportunity with a $<NUM> billion share. MR hardware is likely drive the market, followed by augmented commerce, data, voice, video, and enterprise. As an example, the <NUM> phenomenon of Pokemon Go has truly put forward the advantages of MR in the way people interact with each other and with video games.

Several MR and VR devices have been launched in the last years by companies as Oculus, HTC Vive, Microsoft, Google, Sony, among many others.

As the MR and VR fields mature with improved hardware and software solutions, several companies are now investing on enhancing the user experience by introducing devices with additional sensing and actuation modalities. As an example, several new solutions to enable the sense of touch via haptic sensors and actuators are being investigated and deployed together with MR and VR devices.

In order to further enhance the user experience, solutions providing users with the sense of smell (also known as olfactory sensing) are developed, which can be combined with visual, audio and haptic experiences or experienced on their own.

Digital scent technology is expected to dramatically increase in impact and importance. The companies Scentcom and Inhalió provide a device which releases a wide range of smells to a user using a printer-like dry air cartridge technology. The applications for these devices range from gaming, virtual reality application, e-commerce and automotive. The company Feelreal has developed a VR mask which releases smells. The company Scent Sational Technologies offers various solutions for dispensing smells. On the sensing side, the company Aryballe has created a digital nose to detect and identify odours.

<CIT> discloses an information processing apparatus capable of controlling an odor presentation device according to a usage situation.

<CIT> discloses a television receiver for receiving video and audio signals in addition to a transmission section for converting a composite signal in which a video signal and a smell signal are mixed. The smell signal is generated at an amplifier of a TV camera.

<CIT> discloses an odor communication system in multimedia systems for receiving and sending an odor through a telecommunication line.

It is an object to provide an improved way of sharing olfactory content.

According to a first aspect, it is provided method for communicating olfactory content. The method is performed in an olfactory device and comprises the steps of: obtaining olfactory content to be rendered at a recipient device; identifying an olfactory state at the recipient device; determining an adjustment of at least one olfactory component, based on the identified olfactory state, to render the olfactory content; and triggering the recipient device to perform the adjustment of at least one olfactory component. The steps of identifying an olfactory state, determining an adjustment and triggering are performed for a plurality of recipient devices, based on common olfactory content. The method further comprises the steps, prior to the step of triggering, of: determining a baseline olfactory state based on the olfactory states for each one of the plurality of recipient devices; determining a baseline olfactory adjustment to be made at each recipient device to reach the baseline olfactory state; and triggering each recipient device to perform the baseline olfactory adjustment. The step of determining a recipient olfactory adjustment, the recipient olfactory adjustment is based on determining an olfactory adjustment to be performed in order to change the baseline olfactory state to a state corresponding to the olfactory content.

The step of determining an adjustment may comprise determining the adjustment of the at least one olfactory component to be performed in order to change the olfactory state to a state corresponding to the olfactory content.

The step of determining an adjustment may comprise determining the adjustment by determining a difference in olfactory component values in the olfactory state and in the olfactory content.

The step of identifying an olfactory state may comprise determining an expected olfactory state of a predicted future location of the recipient device.

The step of determining a baseline olfactory state may comprise determining an average of the olfactory states for each one of the plurality of recipient devices.

The step of triggering the recipient device to perform the recipient olfactory adjustment may comprise synchronising the olfactory content with another media content type.

According to a second aspect, it is provided an olfactory device for communicating olfactory content. The olfactory device comprises: a processor; and a memory storing instructions that, when executed by the processor, cause the olfactory device to: obtain olfactory content to be rendered at a recipient device; identify an olfactory state at the recipient device; determine an adjustment of at least one olfactory component, based on the identified olfactory state, to render the olfactory content; and trigger the recipient device to perform the adjustment of at least one olfactory component. The olfactory device further comprises instructions that, when executed by the processor, cause the olfactory device to execute the instructions to identify an olfactory state, determine an adjustment and trigger for a plurality of recipient devices, based on common olfactory content. The olfactory device further comprises instructions that, when executed by the processor, cause the olfactory device to: determine a baseline olfactory state based on the olfactory states for each one of the plurality of recipient devices; determine a baseline olfactory adjustment to be made at each recipient device to reach the baseline olfactory state; and trigger each recipient device to perform the baseline olfactory adjustment. The instructions to determine a recipient olfactory adjustment comprises instructions that, when executed by the processor, cause the olfactory device to determine the recipient olfactory adjustment based on determining an olfactory adjustment to be performed in order to change the baseline olfactory state to a state corresponding to the olfactory content.

The instructions to determine an adjustment may comprise instructions that, when executed by the processor, cause the olfactory device to determine the adjustment of the at least one olfactory component to be performed in order to change the olfactory state to a state corresponding to the olfactory content.

The instructions to determine an adjustment may comprise instructions that, when executed by the processor, cause the olfactory device to determine the adjustment by determining a difference in olfactory component values in the olfactory state and in the olfactory content.

The instructions to identify an olfactory state may comprise instructions that, when executed by the processor, cause the olfactory device to determine an expected olfactory state of a predicted future location of the recipient device.

The instructions to determine a baseline olfactory state may comprise instructions that, when executed by the processor, cause the olfactory device to determine an average of the olfactory states for each one of the plurality of recipient devices.

The instructions to trigger the recipient device to perform the recipient olfactory adjustment may comprise instructions that, when executed by the processor, cause the olfactory device to synchronise the olfactory content with another media content type.

A wireless device may comprise an olfactory device according to the second aspect of the invention.

According to a third aspect, it is provided a computer program for communicating olfactory content. The computer program comprises computer program code which, when run on an olfactory device causes the olfactory device to: obtain olfactory content to be rendered at a recipient device; identify an olfactory state at the recipient device; determine an adjustment of at least one olfactory component, based on the identified olfactory state, to render the olfactory content; and trigger the recipient device to perform the adjustment of at least one olfactory component. The computer program further comprises computer program code which, when run on the olfactory device, causes the olfactory device to identify an olfactory state, determine an adjustment and trigger for a plurality of recipient devices, based on common olfactory content. The computer program further comprises computer program code which, when run on the olfactory device causes the olfactory device to: determine a baseline olfactory state based on the olfactory states for each one of the plurality of recipient devices; determine a baseline olfactory adjustment to be made at each recipient device to reach the baseline olfactory state; and trigger each recipient device to perform the baseline olfactory adjustment. The computer program code, which causes the olfactory device to determine a recipient olfactory adjustment when run on the olfactory device, causes the olfactory device to determine the recipient olfactory adjustment wherein the recipient olfactory adjustment is based on determining an olfactory adjustment to be performed in order to change the baseline olfactory state to a state corresponding to the olfactory content.

The invention will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

Embodiments presented herein improve how olfactory content is shared. This is achieved by considering the olfactory state at the receiver device of the olfactory content. Appropriate adjustments are determined to provide the olfactory content to the recipient.

<FIG> are schematic diagrams illustrating environments in which embodiments presented herein can be applied.

Looking first to <FIG>, there is an olfactory content transmitter <NUM>. The olfactory content transmitter <NUM> can also be the source of the olfactory content <NUM> which is transmitted to a recipient device <NUM>. Olfactory content <NUM> is content which represents odours, analogous to how audio content represents sound, and video content represents (moving) images.

The olfactory content transmitter <NUM> here comprises an olfactory sensor 11a and optionally an olfactory actuator 12a. The olfactory sensor 11a is capable of sensing odours, e.g. from a flower as illustrated, and generates representative olfactory content <NUM> based on the sensed odours. In one embodiment, the odour is broken down into a plurality of olfactory components, where an intensity of each olfactory component is represented by a numerical value. These component values then form part of the olfactory content <NUM>.

Olfactory components are described in more detail in<NPL>. Here, they analysed <NUM> different odours and they realized that the odours consisted of <NUM> different components. For example, an apple odour can be achieved by combining <NUM> different components. The mass spectra of the components were extracted. They then found that there are between <NUM>-<NUM> "basis vectors" (i.e. olfactory components) which can approximate the <NUM> odours. They also showed that one can approximate each of odour using only <NUM> to <NUM> components and still obtain a good classification result in a human evaluation. In another work, the authors compared odours for lime, orange, lemon, mandarin and grapefruit. They show that these odours have several components in common, while other components only exist in some of the odours.

Optionally, instead of detecting odours, the olfactory content transmitter <NUM> extracts odour information by analysing audio and video content which is to be transmitted to the receiver device <NUM>. The olfactory content <NUM> is thus synthetized based on this analysis. For instance, if the olfactory content transmitter <NUM> records video and audio from a barbecue, the olfactory content transmitter can identify the barbecuing and thus synthesize olfactory content representing a barbecue smell.

In another embodiment, the olfactory content transmitter <NUM> allows its user to select an olfactory content to transmit in a message, e.g. in a similar way to how emojis are added to a text message. The selected olfactory content <NUM> then forms part of a message transmitted to the recipient device, without having been sensed by the olfactory content transmitter <NUM>.

Optionally, olfactory content is recorded and associated with a respective location of capture, creating an olfactory content map where different locations are associated with respective olfactory content.

In one embodiment, the smell at the olfactory content transmitter <NUM> is detected for two points in time t1 and t2, where t2 is after t1. The olfactory content <NUM> is then obtained by calculating the difference between the smells at the two points in time. Due to olfactory fatigue, the complete detected current smell (at time t2) may not be what the user intends to share with the other users, but instead, the user intends to share the new smell that has started to be experienced from after time t1. The smell is measured with the olfactory sensor. By basing the olfactory content <NUM> on the difference in smell between t1 and t2, the olfactory state at t1 (which is probably not subjectively experienced by the user at the olfactory content transmitter <NUM> due to olfactory fatigue), the olfactory content <NUM> focuses on the new smell experienced at the olfactory content transmitter.

The olfactory content <NUM> is transmitted from the olfactory content transmitter <NUM> to the recipient device <NUM> via a communication network <NUM>. The communication network <NUM> is any suitable communication network for digital content and can e.g. be any combination of wired and/or wireless networks. For instance, the communication network <NUM> can be a wide area network <NUM>, such as the Internet, optionally also utilising IEEE <NUM>. 11x standards (also known as WiFi) or a cellular network, to allow communication with the recipient device <NUM>. The cellular network can e.g. comply with any one or a combination of LTE (Long Term Evolution), next generation mobile networks (fifth generation, <NUM>), UMTS (Universal Mobile Telecommunications System) utilising W-CDMA (Wideband Code Division Multiplex), CDMA2000 (Code Division Multiple Access <NUM>), or any other current or future wireless network.

The recipient device <NUM> comprises an olfactory actuator 12b and optionally an olfactory sensor 11b. In this embodiment, the recipient device also comprises an olfactory device <NUM>. As explained in more detail below, the olfactory device <NUM> uses the olfactory state at the recipient device <NUM> to determine what olfactory adjustments need to be made at the recipient device to render the olfactory content <NUM>.

The olfactory content transmitter <NUM> and the recipient device <NUM> can be implemented e.g. in the form of a wireless device. The wireless device can be implemented using a smart phone, a tablet computer, laptop, etc. with wireless connectivity. The term wireless is here to be construed as having the ability to perform wireless communication. More specifically, the wireless device can comprise a number of wires for internal and/or external purposes.

<FIG> is a schematic diagram illustrating an environment in which embodiments presented herein can be applied for multiple recipient devices.

In this embodiment, the olfactory content <NUM> is again transmitted from the olfactory content transmitter <NUM>, but is forwarded to several recipients, in this example two recipient devices 2a, 2b. Each one of the recipient devices 2a, 2b comprises an olfactory device <NUM>.

<FIG> is a schematic diagram illustrating an environment in which embodiments presented herein can be applied where the olfactory device <NUM> does not form part of a recipient device. In this example, the olfactory device <NUM> can e.g. be implemented on a server or other suitable computer, and can form part of what is commonly known as 'the cloud'.

<FIG> are flow charts illustrating embodiments of methods for communicating olfactory content <NUM>, wherein the method is performed in the olfactory device of <FIG>. First, the method illustrated by <FIG> will be described.

In an obtain olfactory content step <NUM>, the olfactory device obtains olfactory content to be rendered at a recipient device.

In an identify olfactory state step <NUM>, the olfactory device identifies an olfactory state at the recipient device. The olfactory state can be represented a set of intensities for a plurality of olfactory components.

If the recipient device is implemented using a mask-based device which encloses the nose of the user, the smell measurement is taken within the mask and not outside the mask to properly reflect the olfactory state experienced by the user wearing the recipient device.

When the recipient device is in movement, this step optionally comprises determining an expected olfactory state of a predicted future location of the recipient device. As an example, even though there may be a smell measured at a current location, as the user is walking towards another location, the smell will change and the new smell would be identified e.g. using the olfactory map described above.

In a determine adjustment step <NUM>, the olfactory device determines an adjustment of at least one olfactory component, based on the identified olfactory state, to render the olfactory content. This can comprise determining the adjustment of the at least one olfactory component to be performed in order to change the olfactory state to a state corresponding to the olfactory content.

More specifically, the adjustment can be determined by determining a difference in olfactory component values in the olfactory state and in the olfactory content.

By considering the olfactory state of the recipient device for determining the adjustment, the adjustment can reduce the effects of olfactory fatigue. There are equations and tables which define how to combine certain olfactory components in order to obtain a desired smell, e.g. used in the perfume industry. Such known combinations can be used to generate the olfactory content. The equations and/or tables can be extended to include new combinations for desired olfactory content.

In a trigger recipient adjustment step <NUM>, the olfactory device triggers the recipient device to perform the adjustment of at least one olfactory component.

When the olfactory device is implemented in the recipient device, e.g. as illustrated in <FIG>, this trigger is implemented as an internal signal communication to perform the adjustment.

When the olfactory device is implemented external to the recipient device, e.g. as illustrated in <FIG>, this trigger is implemented as an external signal from the olfactory device to the recipient device via the communication network.

Optionally, this step synchronises the olfactory content with another media content type, such as video and/or audio. For instance, the olfactory content can form part of rich multimedia, where audio, video and smells are all to be rendered for a rich content consumption experience.

In one embodiment, the identify olfactory state step <NUM>, the determine adjustment step <NUM> and the trigger recipient adjustment step <NUM> are performed for a plurality of recipient devices, based on common olfactory content.

Looking now to <FIG>, only new or modified steps compared to the steps of <FIG> will be described.

In an optional determine baseline olfactory state step <NUM>, the olfactory device determines a baseline olfactory state based on the olfactory states for each one of the plurality of recipient devices. The baseline olfactory state can be represented a set of intensities for a plurality of olfactory components.

The baseline olfactory state can e.g. be determined as an average of the olfactory states for each one of the plurality of recipient devices. The average is then calculated individually for each olfactory component.

In an optional determine baseline olfactory adjustment step <NUM>, the olfactory device determines a baseline olfactory adjustment to be made at each recipient device to reach the baseline olfactory state.

In an optional trigger baseline olfactory adjustment step <NUM>, the olfactory device triggers each recipient device to perform the baseline olfactory adjustment. In this way, all the recipient devices are actuated to achieve the common baseline olfactory state. This may increase the user experience since all users subsequently start from the same baseline smell and reduce perception errors when the olfactory content is rendered. Also, this may reduce the amount of smell actuation in future sharing actions, since all recipient devices will share the same baseline. This will reduce actuation energy and use the smell substance.

In one embodiment, the baseline olfactory state is also effected at the olfactory content transmitter. If one of the current recipient devices changes role and acts as a transmitter, no new baseline olfactory adjustment is then required, since all both the previous transmitter and all recipient devices already share the same baseline olfactory state, as long as no new smell disturbance has affected individual recipients in the meantime.

When steps <NUM>, <NUM> and <NUM> are performed, the determine adjustment step <NUM>, comprises determining the recipient olfactory adjustment based on determining an olfactory adjustment to be performed in order to change the baseline olfactory state to a state corresponding to the olfactory content. In such a case, the same adjustment is determined for all recipient devices, since this is based on the baseline olfactory state which is common for all recipient devices.

In one embodiment, if olfactory disturbances (i.e. new smells) occur during the sharing of olfactory content and the baseline of each recipient device is modified, the smell disturbance may be measured and shared among all recipient devices so that the disturbance is compensated. This can also be applied for the baseline olfactory state, where the baseline olfactory state considers the disturbance smell.

<FIG> is a flow chart illustrating embodiments of methods for communicating olfactory content, wherein the method is performed in the olfactory content transmitter of <FIG>.

In a determine source baseline step <NUM>, the olfactory content transmitter determines a baseline olfactory state at the source device. The source baseline can be represented a set of intensities for a plurality of olfactory components.

In a detect change step <NUM>, the olfactory content transmitter detects a change in olfactory state at the source device, the change being greater than a threshold. This can be defined as a change greater than a threshold value in any one olfactory component or an aggregated change of all olfactory components greater than a threshold value.

In a determine olfactory content step <NUM>, the olfactory content transmitter determines olfactory content based on the change in olfactory state. For instance, the olfactory content can be determined as a difference between a sensed olfactory state compared to the baseline olfactory state.

In a transmit olfactory content step <NUM>, the olfactory content transmitter transmits the olfactory content to be rendered at a recipient device.

Using embodiments presented herein, the olfactory fatigue of each user is considered when performing the acquisition and rendering of olfactory content. This results in improved user experience.

An example will now be presented to illustrate an embodiment. A user A is shopping for plants and would like to share the scent of the plant with another user B which is located at home. User A reaches for the smartphone (being the olfactory content transmitter) and brings it close to the plant. User B has a smartphone (being the recipient device) with smell actuation capabilities. The olfactory state has been recorded at the smartphone of user A for a period of time, whereby it is possible to detect the specific smell of the plant as the user brings the smartphone closer to the plant, as the difference in olfactory components.

As both users do not share the same olfactory state, since user A is in a flower shop and user B is at home, a first baseline smell is released at user B's location with the baseline smell of the flower shop. The baseline smell increases the user experience since it allows user B to be immersive in the location of user A. Subsequently, the olfactory content containing the smell detected from the particular plant is released to user B. If, at this stage, user A reaches for another plant, only the smell for the new plant is to be transmitted to User B since they start from the same olfactory state.

<FIG> is a schematic diagram illustrating components of the olfactory device <NUM> and the olfactory content transmitter <NUM> of <FIG> according to one embodiment. A processor <NUM> is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions <NUM> stored in a memory <NUM>, which can thus be a computer program product. The processor <NUM> could alternatively be implemented using an application specific integrated circuit (ASIC), field programmable gate array (FPGA), etc. The processor <NUM> can be configured to execute the methods described above with reference to <FIG> (for the olfactory device) and <FIG> (for the olfactory content transmitter).

The memory <NUM> can be any combination of random access memory (RAM) and/or read only memory (ROM).

An I/O interface <NUM> for communicating with external entities. Optionally, the I/O interface <NUM> also includes a user interface.

Other components are omitted in order not to obscure the concepts presented herein.

<FIG> is a schematic diagram showing functional modules of the olfactory device <NUM> of <FIG> according to one embodiment. The modules are implemented using software instructions such as a computer program executing in the olfactory device <NUM>. Alternatively or additionally, the modules are implemented using hardware, such as any one or more of an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or discrete logical circuits. The modules correspond to the steps in the methods illustrated in <FIG>.

A content obtainer <NUM> corresponds to step <NUM> of <FIG>. A state identifier <NUM> corresponds to step <NUM> of <FIG>. A baseline determiner <NUM> corresponds to step <NUM> of <FIG>. A baseline adjustment determiner <NUM> corresponds to step <NUM> of <FIG>. A baseline adjustment trigger <NUM> corresponds to step <NUM> of <FIG>. A recipient adjustment determiner <NUM> corresponds to step <NUM> of <FIG>. A recipient adjustment trigger <NUM> corresponds to step <NUM> of <FIG>.

<FIG> is a schematic diagram showing functional modules of the olfactory content transmitter <NUM> of <FIG> according to one embodiment. The modules are implemented using software instructions such as a computer program executing in the olfactory content transmitter <NUM>. Alternatively or additionally, the modules are implemented using hardware, such as any one or more of an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or discrete logical circuits. The modules correspond to the steps in the methods illustrated in <FIG>.

A source baseline determiner <NUM> corresponds to step <NUM> of <FIG>. A change detector <NUM> corresponds to step <NUM> of <FIG>. A content determiner <NUM> corresponds to step <NUM> of <FIG>. A content transmitter <NUM> corresponds to step <NUM> of <FIG>.

<FIG> shows one example of a computer program product <NUM> comprising computer readable means. On this computer readable means, a computer program <NUM> can be stored, which computer program can cause a processor to execute a method according to embodiments described herein. In this example, the computer program product is an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. As explained above, the computer program product could also be embodied in a memory of a device, such as the computer program product <NUM> of <FIG>. While the computer program <NUM> is here schematically shown as a track on the depicted optical disk, the computer program can be stored in any way which is suitable for the computer program product, such as a removable solid state memory, e.g. a Universal Serial Bus (USB) drive.

Claim 1:
A method being performed in an olfactory device (<NUM>), the method comprising the steps of:
obtaining (<NUM>) olfactory content to be rendered at a first recipient device (<NUM>, 2a, 2b);
identifying (<NUM>) an olfactory state at a recipient device (<NUM>, 2a, 2b);
determining (<NUM>) an adjustment of at least one olfactory component, based on the identified olfactory state, to render the olfactory content; and
triggering (<NUM>) the recipient device to perform the adjustment of at least one olfactory component;
wherein the step of identifying (<NUM>) an olfactory state, determining (<NUM>) an adjustment and triggering (<NUM>) are performed for a plurality of recipient devices, based on common olfactory content;
characterized in that the method further comprises the steps, prior to the step of determining (<NUM>), of:
determining (<NUM>) a baseline olfactory state based on the identified olfactory states for each one of the plurality of recipient devices;
determining (<NUM>) a baseline olfactory adjustment to be made at each recipient device to reach the baseline olfactory state; and
triggering (<NUM>) each recipient device to perform the baseline olfactory adjustment;
wherein the step of determining (<NUM>) an adjustment of at least one olfactory component is based on determining an olfactory adjustment to be performed in order to change the baseline olfactory state to a state corresponding to the olfactory content.