Patent Description:
The presentation of products presented in stores is often ineffective. Insufficient information is available on the products. Customers get frustrated. On the other hand, it takes a lot of effort from the merchant to find out which products interest the customers to what extent. It is difficult to find out which ways to present products are interesting and which are not, which ways sell best.

<CIT> discloses a system in which an accelerometer detects the movement of a sales article and sends information about it to sales staff, for example. Välkkynen P. (Mobile Augmented Reality for Retail Environments, Proceedings of Workshop on Mobile Interaction in Retail Environments in Conjunction with MobileHCl, <NUM>) shows an augmented reality system intended for mobile use to provide product information to a consumer. <CIT> discloses a vehicle navigation system, whose user interface displays the vehicle as a three-dimensional model. <CIT> discloses a method for combining a virtual 3D model of a body of a person with a 3D image of a garment. <CIT> discloses a method of modelling and showing a virtual jewellery item. <CIT> shows a device utilizing virtual glasses for product presentation. <CIT> discloses a method for displaying an article where augmented reality is used. <CIT> presents pattern recognition strategies and recognition algorithms for recognizing or differentiating objects, and object model database for storing recognition models.

It is an object of the present invention to reduce or even eliminate the abovementioned problems appearing in prior art.

It is an object of the present invention to provide a solution by which familiarisation with the products is pleasant and effective for the customer.

It is an object of the present invention to provide a system by which a retailer, a store chain or product manufacturer can easily and accurately obtain information about which products are of interest to customers and which presentation is effective.

It is an object of the present invention to provide a more efficient way of monitoring the motion and movement of the products presented.

It is an object of the present invention to provide new tools for marketing and sales and for monitoring of them.

It is an object of the present invention to provide new security and logistics solutions.

Among other things, in order to achieve the above-mentioned objects, the system, method and other objects of the invention are characterized by what is set forth in the appended independent claims.

The embodiments, examples and advantages mentioned in this text relate, where applicable, as well to the system, method as to any other aspect of the invention, even though it is not always specifically mentioned.

A typical system according to the invention for presenting physical products to a customer comprises.

A typical method according to the invention for presenting physical products to a customer comprises at least the following steps.

The movement data may comprise, for example, acceleration data produced by an accelerometer, orientation data produced by an orientation sensor or location information produced by a locating device.

For example, a particular department in a store may have multiple products whose movement, acceleration, orientation, or location is monitored. When the system detects a particular product moving, it activates a digital model of that product and begins displaying it on a display device. One display can show several models at the same time. There may be several display devices and they may display the same or different material. For example, each product may be presented on its own display device, or one display may show a digital model that changes with the movement of the product and another display may show other product information related to the product.

In one embodiment of the invention, the monitoring means comprise a plurality of accelerometers attachable to the products for producing acceleration data of the products. In this way, the digital model is activated on basis of the acceleration data produced with the accelerometer attached to the product being moved.

In one embodiment of the invention, the monitoring means comprise pattern recognition means for produce movement data of the products. Thereby, the digital model is activated on basis of the movement data produced with the pattern recognition means.

In one embodiment of the invention, the monitoring means are arranged to monitor the orientation of the products and to produce orientation data of the products. Thereby, the activation means are arranged to activate the digital model of the product being moved on basis of the orientation data produced with the monitoring means.

One embodiment of the invention comprises calculation means, such as a program code, arranged to calculate orientation of the activated digital model on basis of the orientation data, so that the calculated orientation of the digital model changes when orientation of the physical product changes. In this case, in each calculated orientation a certain part of the digital model is calculated to be displayed on the display device. Thereby, the presentation means are arranged to display the activated digital model on the display device in the calculated orientation of the digital model. In this case, the method comprises the following steps:.

In one embodiment, the digital model of the moved product is displayed in the same or corresponding orientation as the moved physical product is in.

In one embodiment of the invention, the data storage comprises product information of the products to be presented, of which product information at least a portion is tied to a specific part of the digital model. Thereby, the system comprises product information selection means, arranged to select and to be displayed on the display device such product information, which is tied to the part of the digital model that is calculated to be displayed on the display device. Said part of the digital model can be, for example, a certain point or certain coordinates in the digital model, e.g. on the surface of the virtual object represented by the model.

In one embodiment of the invention, the digital model of the product comprises two or more two-dimensional images that show the product from different directions. In this case, in each of the calculated orientations, only one of said images is arranged to be displayed on the display device.

A system according to the invention comprises.

An embodiment of the invention comprises on at least one computer.

A method according to the invention of presenting a physical product to a customer comprises at least the following steps.

The required computers, display devices and data transmission devices are known technology as such. A computer typically comprises a memory and program code to be executed therein, the program code arranged to perform required functions. The computer can be a stand-alone device or a server e.g. integrated into the monitoring devices or the display device or used over a network. Data transmission devices may include, for example, transmitters, receivers, conductors, devices required by data network and program codes for using these. Data transmission devices can be wired or wireless. There can be several display devices and they can present the same or different material, such as images, video or text. The display device may be, for example, a computer monitor, a television or a projector and a display surface.

Thus, it has now been found that information about movement, acceleration or orientation of a product can be easily combined with a digital model made of the product. In this way, the model, e.g. an enlarged image of the product, can be presented to a customer interested in the product, for example on a store screen. And when the product is turned, the displayed model can turn at the same pace.

One way of illustrating the invention is to say that with its help a digital model of a physical product presented to a client is activated, where the model is selected from a plurality of models on basis of the product being moved. Thanks to the invention, a digital model of the exact product observed by the customer can. With the invention, the orientation of the activated product can be detected. The invention makes it possible to present a digital model of a product matching to the orientation of the product.

Orientation data can mean, for example, information about the orientation or distance of the product relative to a reference point, such as a floor or a display device, information about turning the product relative to a particular fixed point or coordinate system, or information about movement and location of the product, for example within a store. The orientation data may comprise all or some of the above information.

In one embodiment of the invention, a data storage, such as a database, comprises product information of products, wherein the system comprises product information selection means, such as program code that selects the product information of the moved product to be shown to the customer on a display device. In a corresponding method, the product information of the movable product is displayed to the customer on a display device. The product information may include, for example, technical information describing the product. Product information may include information about restrictions on moving the product and give an alarm on a display or to the personnel if certain limit values are met.

In one embodiment of the invention an alarm is given if the product is handled too hard or in a wrong orientation or if it is taken to a forbidden location.

In one embodiment of the invention, at least a portion of the product information in the data storage is tied to a specific part of the product or the digital model of the product. Thereby, the system comprises means for selecting the product information to be shown to a customer on a display device corresponding to the orientation in which the model is calculated to be displayed. In a corresponding method, product information to be shown to the customer on a display device is chosen according to the orientation in which the model is shown. For example, the display can show technical information of parts of the product that are currently shown on the display.

In one embodiment of the invention, the digital model of the product comprises a three-dimensional model of the product, i.e. a 3D model, such as a CAD drawing. Thereby, calculating means for calculating orientation of the digital model comprise means for turning the 3D model on basis of the orientation data corresponding to the orientation of the physical product. In a corresponding method, when calculating the orientation of the digital model, the 3D model is turned corresponding to the orientation of the physical product. 3D model can mean, for example, a three-dimensional mathematical representation of a surface of a body with help of a specific software.

In one embodiment of the invention, the digital model of the product comprises two or more different images, such as two-dimensional images, that show the product from different directions. In this case, the calculating means for calculating the orientation of the digital model comprise means for selecting an image to be displayed on basis of the orientation information, so that the image as well as possible describes the orientation of the physical product. In a corresponding method, when calculating the orientation of a digital model, the image to be displayed is selected to be the one image that best describes the orientation of the physical product. Changing of the displayed image can be performed e.g. whenever the observed orientation of the physical product surpasses a certain limit value. Images can be, for example, photographs or drawings, such as 2D CAD drawings, from different directions of the product. In a simple example, the model consists of two images, e.g. front side and rear side of the product. When this kind of a system detects on basis of orientation data that the product has been turned beyond a predetermined limit value, for example a "midpoint" level, the image presented on the display device is switched.

In one embodiment of the invention, the digital model of the product comprises one or more video clips, such as a demonstration video, animation, or other motion picture. The digital model may also include sound, light effects, or some other element to capture customer's attention or to entertain them. Different elements can be combined in the same digital model. For example, from one direction or one side of the product the display device may show realistic-looking image of the product but turning the product into a certain orientation can start a video clip.

In one embodiment of the invention, the monitoring means for monitoring orientation of a product and generating orientation data comprise an accelerometer to be connected or attached to the product, and possibly a power supply such as a battery, a microcontroller and a radio transmitter. In a corresponding method, the orientation of the product is monitored, and the orientation data of the product is generated with an accelerometer attached to the product. The accelerometer, power supply, microcontroller and radio transmitter can conveniently be placed in one and the same housing as a so-called orientation sensor that detects a change in its orientation and sends the orientation data forward with the radio transmitter. The accelerometer and other necessary components or the orientation sensor comprising the above-mentioned components can be very small in size. Such a device is easy to attach to almost any product quite discreetly. This way, the customer may not even notice how the orientation of the product she is handling is monitored. The accelerometer may be e.g. of <NUM> or more axis type, depending on the prevailing situation or requirements.

In one embodiment of the invention, an accelerometer or orientation sensor connectable to the product may be integrated in the product to be presented. Several products such as mobile phones, computers or cars have accelerometers or other orientation sensors and transmitters for transmitting the sensor signal. The program codes required by the invention can then be stored in the product in question or its control system. Thus, e.g. an orientation sensor in a mobile phone or in a car can be utilised in the invention. A digital model according to the invention can thus be activated e.g. when a customer grabs a mobile phone, opens a car door or sits in a car.

In one embodiment of the invention, the monitoring means for monitoring orientation of a product and generating orientation data comprise pattern recognition means. In a corresponding method, the orientation of the product is monitored, and the orientation data of the product is produced with pattern recognition means. Pattern recognition means are known and not per se an object of the present invention. They typically comprise a camera and software means for performing a recognition based on an image the camera produces.

In one embodiment of the invention, interaction data, such as information on movement of each product, e.g. interaction count of the product, interaction time, and on the orientation in which the product has been held and for how long, is stored in a data storage, such as a database. This provides measurable data and valuable information, for example, on the effectiveness of product presentations, on the effectiveness between different products, and on possible changes in interaction times and interaction counts due to changes in presentations, offers, and/or other campaigns. In this way, it is possible to estimate how different types of measures affect the handling of products. Changes can be evaluated numerically and analyses can be performed according to KPIs (Key Performance Indicators).

In one embodiment of the invention, in addition to the interaction data collected other measurable information can be utilized, for example information on passers-by and their number, information from door counter, information generated by queueing number machine, and information on checkout reports, e.g. product sales volume and sales times. This makes it possible to evaluate the efficiency of sales funnels for different products.

In one embodiment of the invention, information according to the invention about the products, such as a digital model of a product, is shown on a display device, when a product is moved or when monitoring means such as a pattern recognition device or a motion sensor detect a user to be a particular location, e.g. at a certain distance from a product to be represented. When certain criteria are met, a second information is shown on the display device. The second information could be, for example, a video presentation promoting the product. This kind of a criterion for activating the second information may be, for example, that a certain time has elapsed since any product being monitored has been moved in the vicinity of the display device. This results in a product presentation in which the display device is activated when the customer is interested in the product, that is, e.g. if the customer takes the product in her hand or is present in the vicinity of the product.

The invention is described in more detail below with reference to the enclosed schematic drawing, in which.

For the sake of clarity, same reference numerals are used in different figures and embodiments for some of the corresponding parts.

<FIG>, <FIG> show a consumer <NUM> exploring kitchenware <NUM>, <NUM>, <NUM> on a table <NUM> of a store. An orientation sensor <NUM> is attached e.g. with a sticker to each piece of kitchenware, which orientation sensor is also shown enlarged in <FIG> for clarity.

The corner of the orientation sensor <NUM> has an opening <NUM> for easy attachment. Functionally connected components of the orientation sensor (not shown), i.e. a <NUM>-axis accelerometer, a battery, a BLE radio transmitter (Bluetooth Low Energy) and a microcontroller controlling the operation of the motion sensor, are placed inside a housing <NUM>. Many different wireless transmission technologies can be used in the invention for data transmission, such as BLE, Bluetooth, ZigBee, LoRa. When the accelerometer produces a signal, the radio transmitter transmits it to the receiver <NUM> and further to a central computer <NUM> of the system. The computer <NUM> comprises a media player which controls image and sound to be displayed by a display device <NUM>. Attached on the display device <NUM> is a camera of a pattern recognition device <NUM>. The pattern recognition data storage, such as a database and software, is mainly on the central computer <NUM>. In the memory of the central computer <NUM> there is a data storage such as a database, which maintains digital models <NUM> of the products <NUM>, <NUM>, <NUM>, <NUM> to be presented. Other product information <NUM>, such as technical properties, manuals and price information of the products are also kept in the data storage.

The operation of the system is illustrated in <FIG>. When the customer <NUM> grips a frying pan <NUM>, the accelerometer of the orientation sensor <NUM> generates an orientation data signal <NUM>, which is transmitted wirelessly to the receiver <NUM> by the radio transmitter of the orientation sensor. From there, the orientation data is further directed to the system's central computer <NUM>. The central computer <NUM> comprises software which, after receiving orientation data from the orientation sensor <NUM>, identifies an identifier in the orientation data signal <NUM>, which indicates which product has been moved. In this case, the software activates the digital model <NUM> of the frying pan <NUM> and calculates on basis of the orientation data it received, in which orientation the customer is holding the frying pan <NUM>. The data produced by the software is converted via the media player into a model of the frying pan to be displayed on the display device <NUM>, i.e. into image <NUM>. The model <NUM> of the frying pan to be displayed is in the same orientation as the real frying pan <NUM> is in the customer's hand. In the example of the figures, technical information <NUM>, for example about the materials or the care instructions for the pan, is tied to different parts of the frying pan. This product information is also displayed on the display device <NUM>.

As shown in <FIG>, when the customer <NUM> turns the frying pan <NUM> in his hand, the image <NUM> of the model on the display device also turns. The displayed product information <NUM> is also moved in the image. The product information displayed may also change depending on the orientation of the frying pan <NUM>. The handle <NUM> of the frying pan has moved between the situations in <FIG>, and the product information <NUM> about the handle has moved accordingly. For example, the bottom of the frying pan <NUM> is not shown at all in <FIG>. Product information about the properties of the bottom are also not shown. They are only displayed when the bottom is visible on the display device <NUM>.

Before the customer grabbed the frying pan <NUM>, the display device could show, for example, a video promoting products <NUM>, <NUM>, and <NUM>. After a certain time has elapsed since the customer has lowered the frying pan from his hand to the table <NUM> and has not grasped the other monitored products, i.e. the pot <NUM> or the bowl <NUM>, and has left the vicinity of the table <NUM>, the software of the central computer <NUM> instructs the media player to switch display <NUM> into showing the promoting video again.

The pattern recognition device <NUM> can also be used for monitoring the movements of products according to the invention. The products to be presented and their different orientations may have been taught to the pattern recognition device. The pattern recognition device <NUM> can also be used to monitor customers and their activities. For example, the proximity of the customer can be identified, and sounds or images can be triggered on the display device accordingly.

The information provided by the orientation sensor <NUM> and the pattern recognition device <NUM> can be used to trigger various alarms. For example, if it is detected that a product is dropped or struck too hard, or if inappropriate shaking or throwing of the product is detected, an alarm can be given e.g. to staff. When an alarm is triggered, the camera <NUM> and/or another camera in the store can be turned on, and/or material shot by a camera from a certain period of time can be recorded. If the camera has already shot before the alarm, the footage can be recorded starting e.g. <NUM> seconds or one minute before the alarm was triggered.

Claim 1:
A system for presenting physical products (<NUM>, <NUM>, <NUM>) to a customer (<NUM>), the system comprising
- monitoring means (<NUM>, <NUM>), arranged to monitor movement of the products and to produce movement data of the products;
- a display device (<NUM>);
- at least one computer (<NUM>);
- data transmission means between the monitoring means and the computer and between the computer and the display device;
and on the at least one computer (<NUM>)
- product determining means, arranged to determine which product is moved on basis of the movement data;
- data storage comprising digital models (<NUM>) of a plurality of products (<NUM>, <NUM>, <NUM>);
- activation means arranged to activate a digital model (<NUM>) of the moved product selected from the plurality of models on basis of the movement data produced by the monitoring means (<NUM>, <NUM>);
- presentation means, arranged to display the activated digital model (<NUM>) to the client (<NUM>) on the display device (<NUM>);
characterized in that
the monitoring means are arranged to monitor orientation of the products and to generate orientation data and they comprise a plurality of orientation sensors (<NUM>) each comprising an accelerometer, a power supply, a microcontroller and a radio transmitter, attachable to the products, for producing acceleration data of the products by detecting a change in the orientation sensor's orientation and sending the orientation data (<NUM>) forward with the radio transmitter;
and that the orientation sensors (<NUM>) are arranged to generate an orientation data signal (<NUM>), which is transmitted wirelessly to a receiver (<NUM>) by the radio transmitter of the orientation sensor, from where the orientation data is further directed to the system's computer (<NUM>); the computer (<NUM>) comprising software which, after receiving orientation data from the orientation sensor (<NUM>) is arranged to identify an identifier in the orientation data signal (<NUM>), which indicates which product has been moved.