Patent Description:
Today within the food processing industry, information regarding products are, in accordance with legislation, provided as printed text on the food package. This type of information may be important to the consumer. For instance, it may contain information about the contents of the product which may be important to know in regards to food preference or allergies. It may also contain information regarding expiration date which may be important in regards to the state of the product.

However, this type of information is often difficult to access. It often requires examining multiple sides of the food package before the information has been located. Furthermore, due to the small font it may be written in, it can be hard for people with reduced sight, such as elderly people, to read it.

In light of the above, there is a need for improved ways to present food product information to the consumer such that the consumer can be well aware of what food product that is provided in the package. Zappar, https://web. org/ web/<NUM>/http://www. com/blog/judging-a-product-by-its-cover/, and <NPL> disclose two ways to present food product information to the consumer.

It is an object of the invention to at least partly overcome one or more of the above-identified limitations of the prior art. In particular, it is an object to provide a method for facilitating provision of food product information.

Generally, it has been realized that the use of augmented image data allows for facilitating provision of food product information. Having the possibility to add artificial image data to authentic image data provides namely for that consumers that cannot read, or in any other way not grasp food product information printed on the package, can access this information, which can in turn reduce a risk that a food product containing e.g. nuts is consumed by a consumer with nut allergy.

According to a first aspect it is provided a method for facilitating provision of food product information according to claim <NUM>.

The visual marker should be interpreted as an identifiable marker of a pre-determined size. It could be a simple shape or a complex image.

The food product component should be interpreted as an ingredient of a food product. As a non-limiting example, a food product component may be water, sugar or salt. It may also be in a processed or un-processed form such as either crushed tomatoes or fresh tomatoes.

The artificial image data may also be a text and or text and image in combination. For instance, the artificial image data may comprise an image depicting three oranges, which can be a number of oranges used for producing juice held in the package. Alternatively, the artificial image data may be a text "<NUM> ORANGES" that may be presented next to the package in the augmented image data. Still an option is that the both three oranges and the text is presented next to the package in the augmented image data.

A food product may comprise one food product component, an example of which may be milk. A food product may also comprise multiple food product components, an example of which may be soda, which may comprise the food product components carbonated water, sugar, flavourings and food colouring.

An advantage of the proposed method may be that food product information may be accessed without having to identify where on the package it is written. Furthermore it is possible to visualize the product without having to open the package. Since the food product information is restricted to space available for printing on the food package, this method provides for that that large text that is easy to read can be used also for small portion packages.

The method may further comprise extracting spatial measurement data from the authentic image data, wherein the spatial measurement data is linked to measurements of the food package and/or measurements of the visual marker, generating calibrated artificial image data by performing a spatial calibration of the artificial image data by comparing the spatial measurement data with spatial reference data linked to the food package and/or linked to the visual marker, provided by a spatial reference database, wherein the augmented image data is generated by combining the authentic image data, the calibrated artificial image data and the artificial image spatial reference point.

In other words, the spatial measurement data may be described as geometrical measures of an object determined from an image of the object. In contrast, spatial reference data may be described as geometrical measures determined by measuring the actual object.

An advantage of the steps of generating calibrated artificial data may be that the augmented image data may be scaled to fit with the authentic image data.

The method may further comprise, in the step of providing a food package, providing an additional food package, wherein the additional food package may be provided with an additional visual marker, wherein the authentic image data depicts the visual marker and the additional visual marker, wherein the marker data may be based on the visual marker and the additional visual marker, wherein the artificial image data may depict at least one food product component of a food product held in the food package and at least one additional food product component of an additional food product held in the additional food package, and wherein the artificial image spatial reference point may be based on the visual marker and the additional visual marker.

An advantage of providing an additional package may be that a mix of the food products held in the food packages may be illustrated in the augmented image data.

The spatial measurement data may be linked to measurements of the food package, measurements of the additional food package, measurements of the visual marker and/or measurements of the additional visual marker, and the spatial reference data may be linked to the food package, the additional food package, the visual marker and/or the additional visual marker.

An advantage of this may be that in the case of an additional food package, the augmented image data may be calibrated to both the food package and the additional food package.

The geometric data may be determined by comparing the spatial reference data linked to the food package and/or linked to the visual marker, with the physical container.

The visual marker may be provided on a side of the food package, together with a two-dimensional code, such as a QR-code. This may be advantageous in that the visual marker and the two-dimensional code may be captured in the same image.

The two-dimensional code may be configured to launch an application on the user device. The launch of an application on the user device may be to open a web page in a web browser on the user device. This may be an advantage since no preinstalled application is required.

The two dimensional code may further be configure to give instructions to the user to move the camera backwards, if the code is scanned close to the camera. This way a larger field of view may be obtained.

The two-dimensional code may comprise a unique identifier of the food package. This may be advantageous in that food product information that are relevant to a specific package, such as expiry data, may be provided in the augmented image data.

The visual marker may be comprised in a décor of the food package. The décor of the food package should be interpreted as the printed surface of the package, on which consumer directed information, such as name of the product, is provided. This may be advantageous since no additional marker has to be printed on the food package. In other words, the printing process does not have to be altered. Further, by having less non-consumer directed information, such as a bar code, more space of the package can be used for consumer directed information. Further, by having the visual marker "hidden" in the décor, there is less risk of counterfeit production or other types of fraud. Put differently, by not stating where and how the visual marker is provided on the package, this will be more difficult to copy or unduly amend.

The artificial image data may depict the food product in a first form and the food product may be held in the food package in an second form. A first volume of the food product in the first form may be different from a second volume of the food product in the second form. To depict the food product in different forms may be advantageous since it for instance allows the user to see how full a bowl would be after the food product has been processed, which aids the user in the choice of bowl.

The artificial image data may depend on a geographical position of the user device. This may be an advantage since it may be possible to provide different augmented image data depending on where the food package is located.

The artificial image data may depend on a personal profile of the user. The personal profile may comprise user set data for different food product components. This may be an advantage since it allows for providing food product information relevant to a specific user. For instance it may be food product information regarding possible food product components that the user is allergic to. Another advantage may be that the user may choose what information they want to be presented. For instance they may want to know the contents of the food product and its origin. Or they may want to know the nutrition values and expiry date. To find specific information otherwise would be hard and time consuming.

According to a second aspect, a server configured for facilitating provision of food product information is provided, according to claim <NUM>.

The control unit may further be configured to execute an extracting function configured to extract spatial measurement data from the authentic image data, wherein the spatial measurement data may be linked to measurements of the visual marker, a second generating function configured to generate calibrated artificial image data by performing a spatial calibration of the artificial image data by comparing the spatial measurement data with spatial reference data linked to the visual marker, provided by a spatial reference database.

In addition to the attributes mentioned above, further attributes may be added and used in combination with the attributes mentioned above or separately from the attributes mentioned above.

The same features and advantages described with respect to one aspect are applicable to the other aspects unless explicitly stated otherwise.

<FIG> is a schematic view illustrating a system <NUM> for facilitating provision of food product information by way of example. A food package <NUM> can be provided. The food package <NUM> can hold a food product. The food product can comprise one or more food product components. In this example also an additional food package <NUM> is provided with an additional visual marker <NUM>. It may also be multiple additional food packages. In the case of multiple food packages, two or more food packages may hold the same food product. They may also hold different food products.

A physical container <NUM> is also provided, as illustrated in this example. It can also be multiple physical containers provided. A physical container <NUM> can be a bowl, a pot, a glass, or any other physical container able to hold a food product. The physical container <NUM> may be different from the food package <NUM>.

The food package <NUM> is provided with a visual marker <NUM>. It may also have a two-dimensional code <NUM>, such as a QR-code. The visual marker <NUM> and two-dimensional code <NUM> is further described in regard to <FIG>.

The system <NUM> further comprises a user device <NUM>. The user device can be a mobile device such as a smart phone, or a tablet or any other electronic device fit for the purpose. It can also be smart glasses, wherein a screen is incorporated in a lens of the glasses.

The user device <NUM> can comprise a camera <NUM>. The camera <NUM> can be provided as an external camera, connected to the user device <NUM> by suitable means.

The user device <NUM> comprises a screen <NUM>.

The camera <NUM> of the user device <NUM> can be used to capture authentic image data. The authentic image depicts one or more visual markers <NUM> that are provided on the food package <NUM>. The authentic image data further depicts one or more food packages. It also depicts one or more physical containers.

Marker data <NUM> can be extracted from the authentic image data by the user device <NUM>. The marker data <NUM> is used to retrieve artificial image data <NUM> linked to the marker data <NUM>, from an artificial image database <NUM>. The artificial image database <NUM> can be provided locally on the user device <NUM>. It can also be provided externally to the user device <NUM>. The artificial image data <NUM> depicts food product components that a food product held in the food package is comprised of. In other words, it can be illustrations of the contents of the food package <NUM>. It can also be illustrations or videos related to the food package <NUM>, such as a guide on how to open the food package. It can also be illustrations related to the producer, such as a mascot or logotype of the company.

An artificial image spatial reference point is determined. It can be determined based on a location of the visual marker <NUM>.

Augmented image data can be generated by combining the authentic image data, the artificial image data and the artificial image spatial reference point. The augmented image data can be interpreted as an image showing both real life objects, such as food packages or physical containers, and animated objects from the artificial image database. The artificial image spatial reference point can be used to align the authentic image data and the artificial image data. In a simpler version of the system <NUM>, the step of determining an artificial image spatial reference point can be skipped. Instead, a pre-determined reference point, such as the center of the screen, can be used to know where to place the artificial image data.

The augmented image data is provided to the screen <NUM> of the user device <NUM>.

Optionally, a spatial reference database <NUM> can be provided. The spatial reference database <NUM> can provide spatial reference data <NUM> linked to a food package <NUM> and/or a visual marker <NUM>. The spatial reference data can be geometrical dimensions of the food package or visual marker. To know which package type that is depicted, this information may be linked to the visual marker or other code provided on the package. By knowing which type of package, e.g. Tetra Brik Aseptic <NUM> Slim, spatial reference data linked to the package being depicted can be retrieved. An advantage of using the package for spatial alignment instead of the visual marker is that the package is larger and as an effect different spatial reference points can be placed further apart, which may provide for improved spatial alignment of the artificial image data and the authentic image data.

Spatial measurement data can be extracted from the authentic image data. The spatial measurement data can be linked to measurements of a food package <NUM>, as described above, and/or measurements of a visual marker <NUM>. Calibrated artificial image data can be generated by comparing the spatial measurement data with the spatial reference data.

The augmented image data may be generated using the authentic image data, the calibrated artificial image data and the artificial image spatial reference point. In this way, the size of an artificial object may be calibrated to match the size of the food package <NUM> and/or the visual marker <NUM>.

Geometric data of the physical container <NUM> can be determined, for instance by comparing the physical container <NUM> to spatial reference data of the food package <NUM> and/or the visual marker <NUM>.

<FIG> illustrates, by way of example, a scene based on authentic image data. In this example the food package <NUM> with a visual marker <NUM>, and a physical container <NUM> is visualized.

<FIG> illustrates. By way of example, a scene based on augmented image data. It illustrates the food product held in the food package <NUM> but in the physical container <NUM> in a first form <NUM>. It could also be illustrated in a second form <NUM> as in <FIG>, which shows the same food package <NUM> and the same food product but in a second form <NUM> with a different volume. The first and second form can be interpreted as a processed and un-processed form of the food product.

As an example, the food package <NUM> could be a package of cream. In <FIG> the cream is visualized as un-whipped cream. In <FIG> the cream is visualized as whipped cream. In this case, the first form <NUM> of the product has a smaller volume than the second form <NUM>.

As another example, the food package <NUM> could be a package of orange juice. The first form can be un-processed oranges. The second form can be the processed form of orange juice. In this case, the first form of the product has a larger volume than the second form.

Even though not illustrated in any of <FIG> the augmented image data can depict food products or food product components of multiple food packages in the same scene. This could be used to illustrate a mix of two food products.

As another example, a food product can be illustrated by the food product components it's comprised of separately.

<FIG> is a flowchart illustrating the steps of a method <NUM> for facilitating provision of food product information.

In a first step <NUM>, a food package is provided, wherein the food package is provided with a visual marker.

In a second step <NUM>, authentic image data depicting the visual marker can be captured by a camera of a user device.

In a third step <NUM>, marker data is extracted from the authentic image data.

In a fourth step <NUM>, artificial image data linked to the marker data is retrieved from an artificial image database, wherein the artificial image data depicts at least one food product component of a food product held in the food package.

Optionally, the third and fourth step can be skipped by checking <NUM> if marker data already has been retrieved. In this case the method <NUM> can be repeated continuously, to get a real-time feed of augmented image data on the screen of the user device. In other words, the third step may need to be performed only once. In addition, the fourth step may need to be performed only once.

In a fifth step <NUM>, an artificial image spatial reference point is determined, based on the visual marker.

In a sixth step <NUM>, augmented image data can be generated by combining the authentic image data, the artificial image data and the artificial image spatial reference point.

In a seventh step <NUM>, the augmented image data is provided to a screen of the user device.

Optionally, in an eighth step <NUM>, spatial measurement data from the authentic image data can be extracted, wherein the spatial measurement data can be linked to measurements of the food package and/or measurements of the visual marker.

Optionally, in a ninth step <NUM>, calibrated artificial image data can be generated by performing a spatial calibration of the artificial image data by comparing the spatial measurement data with spatial reference data linked to the food package and/or linked to the visual marker, provided by a spatial reference database.

Optionally, in the sixth step <NUM>, the augmented image data can be generated by combining the authentic image data, the calibrated artificial image data and the artificial image spatial reference point.

Optionally, in the first step <NUM>, an additional food package can be provided <NUM>, wherein the additional food package can be provided with an additional visual marker,.

The spatial measurement data can be linked to measurements of the food package, measurements of the additional food package, measurements of the visual marker and/or measurements of the additional visual marker. Further, the spatial reference data can be linked to the food package, the additional food package, the visual marker and/or the additional visual marker.

In the first step <NUM>, a physical container is provided <NUM>, wherein the authentic image data further depicts the physical container. In a tenth step, geometric data for the physical container in the authentic image data is determined. In an eleventh step, adjusted artificial image data is generated by adjusting the artificial image data based on the geometric data for the physical container. In the sixth step <NUM>, the augmented image data is generated by combining the authentic image data, the adjusted artificial image data and the artificial image spatial reference point.

Optionally, in the tenth step, the geometric data is determined by comparing the spatial reference data linked to the food package and/or linked to the visual marker, with the physical container.

<FIG> illustrates examples of ways the visual marker <NUM> can be provided on the food package <NUM>. The visual marker <NUM> can be provided together with the two-dimensional code <NUM>, such as a QR code, or separately.

<FIG> illustrates an example where the visual marker <NUM> is provided next to the two dimensional code <NUM> on one side of the food package <NUM>. They can be placed on either side of the food package <NUM>.

<FIG> illustrates an example where the visual marker <NUM> is provided within the two-dimensional code <NUM>.

<FIG> illustrates an example where the visual marker <NUM> is comprised in a décor <NUM> of the food package <NUM>. In other words, the visual marker <NUM> can be, for instance, a logo on the package that is already printed on the food package <NUM>.

<FIG> illustrates a schematic view of a server <NUM>. The server <NUM> is configured for facilitating provision of food product information. The server <NUM> comprises a transceiver <NUM>, a control unit <NUM> and a memory <NUM>.

The transceiver <NUM> is configured to communicate with other devices. The transceiver <NUM> can be configured to receive marker data, and transmit augmented image data to a screen of a user device.

The control circuit <NUM> can be configured to perform the control of functions and operations of the server <NUM>. The control circuit <NUM> can include a processor <NUM>, such as a central processing unit (CPU). The processor <NUM> can be configured to execute program code stored in the memory <NUM>, in order to perform functions and operations of the server <NUM>.

The control circuit <NUM> executes a retrieving function <NUM> configured to retrieve artificial image data linked to the marker data from an artificial image database, wherein the artificial image data depicts at least one food product component of a food product held in the food package.

The control circuit <NUM> executes a determining function <NUM> configured to determine an artificial image spatial reference point based on the visual marker.

The control circuit <NUM> can execute a first generating function <NUM> configured to generate augmented image data by combining the authentic image data, the artificial image data and the artificial image spatial reference point.

Optionally, the control circuit <NUM> can execute an extracting function <NUM> configured to extract spatial measurement data from the authentic image data, wherein the spatial measurement data is linked to measurements of the visual marker and/or the package depicted in the authentic image data.

Optionally, the control circuit <NUM> can execute a second generating function <NUM> configured to generate calibrated artificial image data by performing a spatial calibration of the artificial image data by comparing the spatial measurement data with spatial reference data linked to the visual marker and/or the package, provided by a spatial reference database.

The control circuit <NUM> executes a second determining function <NUM> configured to determine geometric data for a physical container in the authentic image data.

The control circuit <NUM> executes a third generating function <NUM> configured to generate adjusted artificial image data by adjusting the artificial image data based on the geometric data for the physical container.

The control circuit <NUM> executes a fourth generating function <NUM> configured to generate the augmented image data by combining the authentic image data, the adjusted artificial image data and the artificial image spatial reference point.

Optionally, the control circuit <NUM> can be configured to extract marker data from the authentic image data, by means of an second extraction function.

The server <NUM> may be provided within the user device.

Claim 1:
A method (<NUM>) for facilitating provision of food product information, comprising;
providing (<NUM>) a food package (<NUM>), wherein the food package (<NUM>) is provided with a visual marker (<NUM>), (<NUM>),
providing (<NUM>) a physical container (<NUM>), different from the food package (<NUM>),
capturing (<NUM>), by a camera (<NUM>) of a user device (<NUM>), authentic image data depicting the visual marker (<NUM>) and the physical container (<NUM>),
extracting (<NUM>) marker data (<NUM>) from the authentic image data,
retrieving (<NUM>) artificial image data (<NUM>) linked to the marker data (<NUM>) from an artificial image database (<NUM>), wherein the artificial image data (<NUM>) depicts at least one food product component of a food product held in the food package (<NUM>),
determining (<NUM>) an artificial image spatial reference point based on the visual marker (<NUM>),
determining geometric data for the physical container (<NUM>) in the authentic image data,
generating adjusted artificial image data by adjusting the artificial image data (<NUM>) based on the geometric data for the physical container (<NUM>),
generating (<NUM>) the augmented image data by combining the authentic image data, the adjusted artificial image data and the artificial image spatial reference point, and
providing (<NUM>) the augmented data to a screen (<NUM>) of the user device (<NUM>).