Patent Description:
With the development and improvement of image recognition technology, image recognition technology is widely applied in various industries. In the commodity retail business, intelligent management of shelf commodities may be realized by capturing products on the shelves and performing image recognition on the captured images.

There have been several patents issued that cover variations of the goods information obtaining device and the shelf managing system. For example, US Patent Application Publication No. <CIT> discloses a shelf according to the preamble of claim <NUM> with a reflection based tracking system comprising a processor. The processor can receive data representing a view reflected by a mirror of a plurality of mirrors. The plurality of mirrors may be configured in a space to reflect a plurality of views of structures in the space. The mirror of the plurality of mirrors may include a uniquely identifiable feature distinguishable from other objects in the space. The processor can identify the mirror of the plurality of mirrors according to the uniquely identifiable feature. The processor can also determine an attribute of the structures according to the identified mirror and the data representing the view reflected by the mirror.

The invention provides a shelf according to claim <NUM>. The goods information obtaining device of said shelf includes: at least one reflector; and at least one image sensor for sensing a reflection image of goods within the shelf unit that are reflected one or more times by the at least one reflector.

In some embodiments, each of the at least one reflector includes a plane mirror or a convex mirror.

In some embodiments, the at least one image sensor includes a plurality of image sensors, and viewing angles of lenses of the plurality of image sensors adjoin or partially coincide each other at a coverage area of the at least one reflector.

In some embodiments, the goods information obtaining device further includes a driver for driving the at least one image sensor or the reflector to move, such that a viewing angle of a lens of each of the at least one image sensor covers reflection images of different portions of the goods on the at least one reflector at different moments.

The shelf includes: a shelf unit for containing goods; and the goods information obtaining device previously described.

The at least one reflector is located on one side of a top of the shelf unit close to the goods, and each of the at least one image sensor includes a lens, wherein the lens has a viewing angle covering at least a portion of the at least one reflector.

The lens has a height no higher than a height of the goods.

The height of the lens is configured to be flush with a bottom plane of the shelf unit.

In some embodiments, the shelf unit has an open end for accessing the goods, and the at least one image sensor is located within the shelf unit on one side of the shelf unit away from the open end.

In some embodiments, the shelf unit further has a closed end, where the at least one image sensor is located.

In some embodiments, a minimum distance d between the lens and the goods satisfies: <MAT> where W is a maximum distance between the goods and the lens, H is a height of the goods, and D' is a minimum spatial height of the shelf unit.

In some embodiments, the at least one reflector includes: a first reflector, located on one side of a top of the shelf unit close to the goods; and a second reflector, located on one side opposite to the first reflector; wherein each of the at least one image sensor includes a lens having an orientation configured such that a viewing angle of the lens covers at least a portion of the second reflector.

In some embodiments, which are not a part of the present invention, the at least one reflector is located external to the shelf unit.

In another aspect, a shelf managing system according to claim <NUM> is provided. The shelf managing system includes:
the shelf previously described; and a processor for processing the reflection image.

In some embodiments, the processor is configured to perform at least one of the following operations: analyzing the reflection image to obtain goods information related to placement within the shelf unit; stitching reflection images to form a complete reflection image of the goods within the shelf unit; or detecting whether the goods placed within the shelf unit are out of stock or are wrong goods according to the reflection image.

In another aspect, a shelf managing method according to claim <NUM> is provided. The shelf managing method includes: sensing by an image sensor a reflection image of goods within a shelf unit that are reflected one or more times by at least one reflector; analyzing the reflection image to obtain goods information related to placement within the shelf unit.

In some embodiments, the shelf managing method further includes: judging whether the goods placed within the shelf unit are out of stock or are wrong goods according to the goods information related to placement.

In some embodiments, the shelf managing method further includes: stitching reflection images to form a complete reflection image of the goods within the shelf unit.

The accompanying drawings, which constitute part of this specification, illustrate exemplary embodiments of the present disclosure and, together with this specification, serve to explain the principles of the present disclosure.

The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:.

Embodiments of the invention become apparent from the dependent claims. The examples shown in <FIG>,<FIG> and <FIG> do not from part of the present invention.

It should be understood that the dimensions of the various parts shown in the accompanying drawings are not drawn according to the actual scale. In addition, the same or similar reference signs are used to denote the same or similar components.

Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. The description of the exemplary embodiments is merely illustrative and is in no way intended as a limitation to the present disclosure, its application or use.

The use of the terms "first", "second" and similar words in the present disclosure do not denote any order, quantity or importance, but are merely used to distinguish among different parts. Such similar words as "including" or "containing" mean that the element preceding the word encompasses the elements enumerated after the word, and does not exclude the possibility of encompassing other elements as well. "Up", "down", "left", "right", and the like are used only to present relative positional relationships, which may also be possibly changed correspondingly when a change is made to an absolute position of the described object.

In the present disclosure, when it is described that a particular device is located between the first device and the second device, there may be an intermediate device between the particular device and the first device or the second device, and alternatively, there may be no intermediate device. When it is described that a particular device is connected to other devices, the particular device may be directly connected to said other devices without an intermediate device, and alternatively, may not be directly connected to said other devices but with an intermediate device.

All the terms (including technical and scientific terms) used in the present disclosure have the same meanings as understood by those skilled in the art of the present disclosure unless otherwise defined.

In some related arts, a camera for capturing the products is provided within the shelf. Moreover, in order to enable the camera to capture more product images, the camera often needs to be set at a high position, for example at the top of each layer of shelves, so as to capture the products placed on the bottom plate of the layer of shelves. The inventors have found after research that the camera in the related art needs to be set at a high position, so that the spacing between layers of the shelves is correspondingly increased, which causes that the space utilization rate of the shelf drops.

In view of this, embodiments of the present disclosure provide a goods information obtaining device, a shelf, a shelf managing system, and a method, which can allow the shelf unit to have a smaller spatial height.

<FIG> is a schematic structural view according to one embodiment of the shelf of the present disclosure.

Referring to <FIG>, in some embodiments, the shelf <NUM> includes: a shelf unit and a goods information obtaining device. The shelf unit has a containing space which may be used for containing the goods <NUM> so as to display or sell the goods <NUM>. In <FIG>, the overall structure of the shelf <NUM> may be formed by assembling an upright post <NUM>, a vertical partition <NUM> and a horizontal partition <NUM>. The vertical partition <NUM> and the horizontal partition <NUM> may enclose one or more shelf units. In other embodiments, the shelf <NUM> may also use other structural forms, such as an integrated or split shelf.

In some embodiments, the shelf <NUM> may include a single shelf unit. In other embodiments, the shelf <NUM> can also include a plurality of shelf units, which may be distributed in a single shelf layer or a plurality of shelf layers. In addition, other members such as a readable/writable label <NUM> and the like may be provided in the shelf <NUM> as needed.

The goods information obtaining device includes at least one reflector <NUM> and at least one image sensor <NUM>. The image sensor <NUM> is used for sensing a reflection image <NUM>' of the goods <NUM> within the shelf unit, wherein the goods are reflected one or more times by the at least one reflector <NUM>.

The image sensor <NUM> senses the reflection image <NUM>' of the goods <NUM> within the shelf unit. The image sensor <NUM> may include a Charged Coupled Device (referred to as CCD for abbreviation) image sensor or a Complementary Metal Oxide Semiconductor (referred to as CMOS for abbreviation) image camera of or the like. The reflection image <NUM>' may come from the reflection of the goods <NUM> at the reflector, such as a mirror or other objects capable of providing a reflective surface. The reflection image <NUM>' here may be a reflection image that is reflected once or a reflection image that is reflected for multiple times.

As shown in <FIG>, the reflector <NUM> is disposed within the shelf unit to provide a source for the reflection image to the image sensor <NUM>. The image sensor <NUM> is capable of sensing reflection image <NUM>' of the goods <NUM> within the shelf unit that is reflected once by the reflector <NUM>. The reflector <NUM> may include a plane mirror to form a clear and regular reflection image <NUM>', thereby reducing the difficulty in subsequent analysis and processing of the reflection images <NUM>'. In other embodiments, the reflector <NUM> may also include a convex mirror. With a convex reflective mirror, it is possible to allow the image sensor <NUM> to obtain a wider range of goods reflection image.

As shown in <FIG>, the reflector <NUM> is disposed on one side of a top of the shelf unit close to the goods, so as to adequately utilize the spatial height within the shelf unit. In some embodiments, not being part of the present invention, the reflector <NUM> may be disposed in the form of an independent member below the bottom plate of the upper layer of shelf units. In other embodiments, not being part of the present invention, the reflector <NUM> may be mounted on the lower surface of the bottom plate of the upper layer of shelf units, or the reflective surface may be machined on the lower surface of the bottom plate of the upper layer of shelf units to form the reflector <NUM> or the like.

Referring to <FIG>, the image sensor <NUM> includes a lens. The lens has an orientation configured such that a viewing angle of the lens covers at least a portion of the reflector <NUM>. When the reflector <NUM> is disposed on the top inside the shelf unit, the lens of the image sensor <NUM> is directed upward or obliquely upward so that the viewing angle of the lens can cover a part or entirety of the reflector <NUM>. The image sensor <NUM> is disposed inside the shelf unit.

By collecting the reflection image of the goods within the shelf unit, the above-described embodiments of the goods information obtaining device and the shelf of the present disclosure may reduce the setting height of the image sensor relative to the bottom surface of the shelf unit, thereby allowing the shelf unit to use a smaller spatial height, and further improving the overall space utilization rate of the shelf.

In order to explain the saving effect of the shelf according to the embodiments of the present disclosure in the spatial height compared to the shelf in the related art, a comparative description will be made with reference to <FIG> subsequently.

<FIG> is a schematic view of a viewing angle coverage of an image sensor according to the related art. <FIG> is a schematic view of a forward viewing angle coverage of an image sensor according to one embodiment that is not a part of the present invention.

In <FIG>, the image sensor <NUM> is disposed at the top of the shelf unit and its lens is provided downward to capture the goods <NUM> placed within the shelf unit. Assuming that the maximum viewing angle of the lens of the image sensor is θ, in order to obtain a complete top image of the goods <NUM>, the minimum spatial height of the shelf unit is D, the height of the goods <NUM> is H, and the spatial length occupied by the goods <NUM> in the forward view is L. D satisfies the following relationship: <MAT>.

Suppose θ is approximately <NUM>°, then <MAT>, i.e. <MAT>. The minimum spatial height D of the shelf unit should be greater than the height H of the goods <NUM>, i.e. D>H.

According to the formula (<NUM>), it may be calculated that D satisfies: <MAT>.

Referring to <FIG>, in some embodiments of the shelf not being part of the present invention, in order to avoid that the image sensor <NUM> is shielded by the goods <NUM>, the height of the lens may be configured to be no less than that of goods <NUM>. If the image sensor <NUM> is set higher, it is also possible to increase the spatial height of the shelf along therewith. For convenience of calculation, the height of the lens of the image sensor <NUM> is set to be the same as the height of the goods <NUM> in <FIG>.

Referring to <FIG>, it is still assumed that the maximum viewing angle of the lens of the image sensor is θ. Since the height of the lens of the image sensor <NUM> is set to be the same as the height of the goods <NUM>, so that there is no such a problem that the image sensor <NUM> is shielded by the goods <NUM>, the actual viewing angle of the image sensor is also θ. The minimum spatial height of the shelf unit is D', and in order to obtain a complete top image of the goods <NUM>, the height of the goods <NUM> is H. The spatial length occupied by the goods <NUM> in the forward view is L. D' satisfies the following relationship:.

According to the formula (<NUM>), it may be derived that D' satisfies: <MAT>.

If the formula (<NUM>) is substituted into the formula (<NUM>), it may be obtained that:<MAT>.

Since D>H, D'<D. Thus, it may be known that the shelf unit of the present embodiment can allow to use a smaller spatial height than the shelf in the related art, thereby effectively improving the space utilization rate of the shelf.

<FIG> is a schematic view of a side viewing angle coverage of an image sensor according to one embodiment of the shelf not being part of the present invention.

Referring to <FIG>, the shelf unit has an open end for accessing the goods <NUM> (i.e., an open area on the right side of the goods), and the image sensor <NUM> may be disposed within the shelf unit and on one side away from the open end (i.e., the left side of the goods). In this way, the goods <NUM> may be placed close to the open end so as to be accessed or placed by the purchaser or the staff responsible for loading and unloading the goods with convenience. The side may be set to be a closed end, and the image sensor <NUM> may be fixed at the closed end or fixed at a position close to the closed end. In other embodiments, the installation position of the image sensor <NUM> may also be an open end.

In <FIG>, the height of the lens of the image sensor <NUM> may be set to be the same as the height of the goods <NUM>. In some application scenarios, it is possible to maintain a certain distance between the goods <NUM> and the image sensor <NUM> on the other side of the goods <NUM> when the goods <NUM> is placed. Thus, even if the height of the lens of the image sensor <NUM> is set lower, it is also possible to avoid that the lens is shielded by the goods <NUM>. Consequently, depending on the placement distance of the goods relative to the image sensor <NUM>, the height of the lens may be configured to be no higher than the height of the goods. Correspondingly, the allowed spatial height of the shelf unit of the shelf <NUM> may be further reduced.

<FIG> is a schematic view of a forward viewing angle coverage of an image sensor according to another embodiment of the shelf of the present disclosure.

Referring to <FIG>, according to the invention, the height of the lens is configured to be flush with a bottom plane of the shelf unit. If it is still assumed that the maximum viewing angle of the lens of the image sensor is θ, the vertical distance S' between the reflection image of the goods <NUM> and the lens of the image sensor <NUM> is D'+(D'-H). Moreover, the vertical distance S between the reflection image of the lens of the image sensor <NUM> and the goods <NUM> in the related art shown in <FIG> is D-H. When the maximum viewing angle of the lens of the image sensor is θ, and the spatial length of the goods <NUM> occupied in the forward view is L, referring to the formula (<NUM>), it may be derived that S'=S=(tan(θ /<NUM>)*L)/<NUM>, i.e. D'+(D'-H)=D-H, so that it may be derived that D'=D/<NUM>. Currently, the minimum spatial height D' of the shelf unit in this embodiment is only half of the minimum spatial height D of the shelf unit in the related art, thereby effectively improving the space utilization rate of the shelf.

<FIG> is a schematic view of a side viewing angle coverage of an image sensor according to another embodiment of the shelf of the present disclosure.

Referring to <FIG>, the image sensor <NUM> currently is already substantially at the lowest position of the shelf unit. Considering that the goods <NUM> might shield the viewing angle of the lens of the image sensor <NUM>, the actual viewing angle of the lens of the image sensor <NUM> is θ'/<NUM> after the lens is shielded by the goods <NUM> on the right side, and θ' is required to be less than or equal to the maximum viewing angle θ of the camera.

In order to eliminate the effect that the actual viewing angle is shielded, when the goods <NUM> is placed, the minimum distance d between the goods <NUM> and the image sensor <NUM> may satisfy that: <MAT> and at the same time satisfies that:
W is a furthest distance between the goods <NUM> and the image sensor <NUM>, H is a height of the goods <NUM>, and D' is a minimum spatial height of the shelf unit. According to the formulas (<NUM>) and (<NUM>), it may be derived that the minimum distance d between the goods <NUM> and the image sensor <NUM> is: <MAT>.

The staff may set the position of the image sensor <NUM> according to the formula (<NUM>), or place the goods <NUM> according to the formula (<NUM>), so as to effectuate covering the complete reflection image of the articles within the shelf unit whilst effectuating a small allowed spatial height of the shelf unit.

<FIG> is a schematic view of an image sensor sensing reflection image reflected for multiple times according to one embodiment of the shelf that is not a part of the present invention.

Referring to <FIG>, a plurality of reflectors <NUM>, such as a first reflector <NUM> and a second reflector <NUM>, may be disposed within the shelf unit. The first reflector <NUM> is located on one side of a top of the shelf unit close to the goods <NUM>; and the second reflector <NUM> is located on one side opposite to the first reflector <NUM>. The image sensor <NUM> includes a lens. The lens has an orientation configured such that a viewing angle of the lens covers at least a portion of the second reflector <NUM>.

In <FIG>, the first reflector <NUM> having a large size may be disposed inside the shelf unit and on the top of the shelf unit, and the second reflector <NUM> having a small size may be disposed on one side close to a position where the image sensor <NUM> is located. The viewing angle of the lens of the image sensor <NUM> may cover the second reflector <NUM> and sense the reflection image <NUM>" reflected by the second reflector <NUM>. The reflection image <NUM>" is derived from the reflection effect of the second reflector <NUM> over the reflection image <NUM>' and the reflection image <NUM>' is derived from the reflection effect of the first reflector <NUM> over the goods <NUM> below the same. In this way, the image sensor <NUM> can sense reflection image <NUM>" of the goods <NUM> within the shelf unit that is reflected for multiple times by the plurality of reflectors <NUM>. By sensing the reflection image that is reflected for multiple times, it is possible to further reduce the allowed space of the shelf unit of the shelf and cover a larger placement range of the goods. On the other hand, by configuring a plurality of reflectors at different positions, the image sensor <NUM> may be set at more flexible positions.

In some embodiments of the shelf presented above, the viewing angle of the lens of the image sensor exactly covers the reflection image of all the goods of the shelf unit for ease of understanding and formula derivation. However, in actual use, it is possible to allow the viewing angle of the lens of the image sensor to cover the reflection image of all the goods of the shelf unit, or it is also possible to only cover the reflection image of part of the goods in the shelf unit.

<FIG> is a schematic view of a viewing angle coverage of an image sensor according to one embodiment of the goods information obtaining device that is not a part of the present invention.

Referring to <FIG>, the goods information obtaining device may include a plurality of image sensors 12a, 12b , and viewing angles of lenses of the plurality of image sensors may respectively cover a partial area of the reflector <NUM>. Moreover, the coverage areas respectively corresponding to these image sensors 12a, 12b may adjoin with each other or partially coincide. The reflection images <NUM>' of the goods <NUM> collected by the image sensors 12a, 12b respectively, may form an overall reflection image of the goods within the shelf unit in an image stitching manner. Alternatively, the reflection images <NUM>' of the goods <NUM> collected by the image sensors 12a, 12b respectively, may be subjected to image processing respectively.

By providing a plurality of image sensors, even if the viewing angle of the lens of each image sensor can cover a limited range of reflection image, it is still possible to obtain complete goods information related to placement in a subsequent image processing manner. In addition, referring to the previous calculation formulas, when the spatial height of the shelf unit is further reduced, the collection of the reflection image may be performed by providing more image sensors, so that it is possible to allow a spatial height of a smaller shelf unit.

In addition to the manner of providing a plurality of image sensors, the image sensors may also be driven to move to achieve a wider range of coverage.

<FIG> is a schematic view of a viewing angle coverage of an image sensor according to another embodiment of the goods information obtaining device according to the present disclosure.

Referring to <FIG>, in some embodiments, the goods information obtaining device further includes a drive mechanism. The drive mechanism may be a motor, a hydraulic cylinder, an air cylinder or a pulley block and the like. The driver is used for driving the image sensor <NUM> to move, such that a viewing angle of the lens of the image sensor <NUM> covers reflection images <NUM>' of different portions of the goods <NUM> on the reflector <NUM> at different moments. The reflection images <NUM>' of the goods <NUM> collected by the image sensors <NUM> at different moments, may form an overall reflection image of the goods within the shelf unit in an image stitching manner. Alternatively, the reflection images <NUM>' of the goods <NUM> collected by the image sensors <NUM> at different moments, may be subjected to image processing respectively. In addition to driving the image sensors <NUM> to move, the drive mechanism may also drive the reflector <NUM> to move such as translate or rotate the reflector <NUM>. Of course, the drive mechanism may also drive both the reflector <NUM> and the image sensors <NUM> to move.

By the driver driving the image sensor or the reflector, even if the viewing angle of the lens of the image sensor can cover a limited range of reflection image, it is still possible to obtain complete goods information related to placement in a subsequent image processing manner. In addition, referring to the previous calculation formulas, when the spatial height of the shelf unit is further reduced, the drive mechanism may be correspondingly provided to achieve a greater range of motion, thereby allowing a spatial height of a smaller shelf unit.

In other embodiments, the drive mechanism may also be used for driving a plurality of image sensors or a plurality of reflectors to move.

The above-described shelves may be applied in various fields or occasions where the acquisition of the shelf placement information is required, for example the smart shelf field. The present disclosure also provides some embodiments of a shelf management system based on the above-described respective shelf embodiments.

<FIG> is a block schematic diagram according to one embodiment of the shelf managing system according to the present disclosure.

Referring to <FIG>, in some embodiments, the shelf managing system includes any of the foregoing embodiments of the goods information obtaining device <NUM> and a processor <NUM>. The processor <NUM> is used for processing the reflection image of the goods within the shelf unit. The reflection image is derived from the sensed data of the image sensor <NUM> in the goods information obtaining device <NUM>.

In some embodiments, the processing operations of the reflection image <NUM> by the processor may include: analyzing the reflection image to obtain goods information related to placement within the shelf unit. For example, the reflection image is compared with the standard images in the database to analyze whether the current goods placement is normal. For another example, the image feature information is extracted from the reflection image or the reflection image is semantically analyzed to determine the current goods information related to placement and the like.

The processing operations of the reflection image <NUM> by the processor may also include: detecting whether the goods placed within the shelf unit are out of stock or are wrong goods according to the reflection image. For example, when the reflection image is inconsistent with the standard placed images in the database, it is indicated that the shelf unit is out of stock or contains wrong goods and needs to be adjusted by the staff.

In other embodiments, the processing operations of the reflection image <NUM> by the processor may also include: stitching reflection images to form a complete reflection image of the goods within the shelf unit. The complete reflection image of the goods within the shelf unit is formed by using an image stitching method in the related art, in which for example, the reflection images sensed by a plurality of image sensors respectively or a plurality of reflection images collected by the image sensors at different moments are subjected to stitching by a processor.

<FIG> is a block schematic diagram according to another embodiment of the shelf managing system according to the present disclosure.

Referring to <FIG>, in some embodiments, the shelf managing system includes any of the foregoing embodiments of the shelf <NUM> and a processor <NUM>. The processor <NUM> is used for processing the reflection image of the goods within the shelf unit. The reflection image is derived from the sensed data obtained by the goods information obtaining device <NUM>.

In other embodiments, the processing operations of the reflection image <NUM> by the processor may also include: stitching reflection images to form a complete reflection image of the goods within the shelf unit. The complete reflection image of the goods within the shelf unit are formed by using an image stitching method in the related art, in which for example, the reflection images sensed by a plurality of image sensors respectively or a plurality of reflection images collected by the image sensors at different moments are subjected to stitching by a processor.

<FIG> is a schematic flow chart according to one embodiment of the shelf managing method according to the present disclosure.

Referring to <FIG>, in some embodiments, the shelf managing method may include a step <NUM> and a step <NUM>. The shelf managing method may be implemented based on any one of the foregoing embodiments of the shelf. In step <NUM>, a reflection image of goods within a shelf unit that are reflected one or more times by an at least one reflector is sensed by an image sensor. In step <NUM>, the reflection image is analyzed to obtain goods information related to placement within the shelf unit.

Claim 1:
A shelf, comprising:
a shelf unit for containing goods (<NUM>); and a goods information obtaining device (<NUM>) for obtaining goods information within the shelf unit,
wherein the goods information obtaining device comprises at least one reflector (<NUM>) and at least one image sensor (<NUM>) for sensing a reflection image (<NUM>'; <NUM>") of goods (<NUM>) within the shelf unit that are reflected one or more times by the at least one reflector (<NUM>), wherein each of the at least one image sensor (<NUM>) comprises a lens, wherein the lens has a viewing angle covering at least a portion of the at least one reflector (<NUM>), characterized in that the at least one reflector (<NUM>) is located on one side of a top of the shelf unit close to the goods (<NUM>), wherein the height of the lens is configured to be flush with a bottom plane of the shelf unit.