EXHIBIT 1

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 1 of 19 PageID #: 21
US011087272B2 

( 12 ) United States Patent 

Skaff et al . ( 10 ) Patent No .: US 11,087,272 B2 

( 45 ) Date of Patent : Aug. 10 , 2021 

( 54 ) SYSTEM AND METHOD FOR LOCATING , IDENTIFYING AND COUNTING ITEMS ( 58 ) Field of Classification Search 

??? G06Q 10/087 ; G06Q 10/00 ; G06Q 30/02 ; 

G06Q 30/0603 ; GO5D 1/0261 ; 

( Continued ) ( 71 ) Applicant : Bossa Nova Robotics IP , Inc. , Pittsburgh , PA ( US ) 

( 56 ) References Cited 

U.S. PATENT DOCUMENTS 

( 72 ) Inventors : Sarjoun Skaff , Pittsburgh , PA ( US ) ; 

Jonathan Davis Taylor , Pittsburgh , PA 

( US ) ; Stephen Vincent Williams , Pittsburgh , PA ( US ) ; Simant Dube , Berkeley , CA ( US ) 9,097,800 B1 * 

10,007,964 B1 * 

10,169,677 B1 * 

8/2015 Zhu 

6/2018 Calhoon 

1/2019 Ren 

( Continued ) 

GO1S 7/4802 

G06K 9/4604 

HO4W 4/33 ( 73 ) Assignee : BOSSA NOVA ROBOTICS IP , INC . , Leetsdale , PA ( US ) 

FOREIGN PATENT DOCUMENTS ( * ) Notice : Subject to any disclaimer , the term of this 

patent is extended or adjusted under 35 

U.S.C. 154 ( b ) by 332 days . JP 

WO 

2014170431 A 

2014087725 A1 

9/2014 

6/2014 

( 21 ) Appl . No .: 15 / 471,722 OTHER PUBLICATIONS 

( 22 ) Filed : Mar. 28 , 2017 “ Photographing an image of display shelfs while pausing ” and 

“ Thorough audit of store based on 3D map ” . ( 65 ) Prior Publication Data 

US 2017/0286901 A1 Oct. 5 , 2017 

Related U.S. Application Data 

( 60 ) Provisional application No. 62 / 314,785 , filed on Mar. 29 , 2016 , provisional application No. 62 / 427,509 , filed on Nov. 29 , 2016 . 

Primary Examiner Florian M Zeender 

Assistant Examiner Vanessa Deligi 

( 74 ) Attorney , Agent ,or Firm - David R. Stevens ; 

Stevens Law Group 

( 51 ) Int . Ci . GO6Q 10/08 GO6K 9/62 

( 2012.01 ) 

( 2006.01 ) 

( Continued ) 

( 57 ) ABSTRACT 

A system for building a product library without requiring a 

planogram includes an image capture unit operated to pro vide images of items . Taking data from a shelf label detector 

and depth map creation unit , a processing module can be 

used to compare detected shelf labels to a depth map , define 

a product bounding box , and associate the bounding box 

with an image provided by the image capture unit to build 

( Continued ) 

( 52 ) U.S. CI . CPC G06Q 10/087 ( 2013.01 ) ; G05D 1/0246 

( 2013.01 ) ; GO6K 9/00201 ( 2013.01 ) ; 

( Continued ) 

200 

22 

230 

205 202 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 2 of 19 PageID #: 22
US 11,087,272 B2 

Page 2 

image descriptors . The system can include one or more ( 56 ) References Cited 

autonomous robots for pporting and moving the image 

capture unit and other components . U.S. PATENT DOCUMENTS 

21 Claims , 7 Drawing Sheets 

( 51 ) Int . Ci . GO6N 20/00 ( 2019.01 ) 

G05D 1/02 ( 2020.01 ) 

GO6K 9/32 ( 2006.01 ) 

GO6K 9/00 ( 2006.01 ) 

( 52 ) U.S. Ci . CPC G06K 9/00664 ( 2013.01 ) ; G06K 9/3216 

( 2013.01 ) ; G06K 9/3241 ( 2013.01 ) ; G06K 

9/6267 ( 2013.01 ) ; G06N 20/00 ( 2019.01 ) 

( 58 ) Field of Classification Search 

CPC GO5D 1/0274 ; GO5D 1/0246 ; G05D 

2201/0216 ; G07G 1/12 ; G06K 9/00201 ; 

G06K 9/00664 ; G06K 9/3216 ; G06K 

9/3241 ; G06K 9/6267 

See application file for complete search history . 

2008/0077511 A1 * 3/2008 Zimmerman G06Q 10/00 

705/28 

2009/0059270 A1 3/2009 Opalach et al . 2009/0063307 A1 3/2009 Groenovelt et al . 2009/0094140 A1 4/2009 Kwan 

2010/0065634 A1 * 3/2010 Nakamura G06Q 30/0603 

235/385 

2013/0010103 A1 * 1/2013 Ihara G06Q 30/06 

348/116 

2013/0182114 Al 4/2013 Zhang et al . 2014/0003655 A1 1/2014 Gopalakrishnan et al . 2014/0003727 Al 1/2014 Lortz et al . 2015/0139536 Al 5/2015 Jin et al . 2015/0363758 A1 * 12/2015 Wu G06Q 10/087 

705/20 

2016/0180533 Al 3/2016 Pavani et al . 2016/0155011 A1 * 6/2016 Sulc GO6K 9/00671 

382/103 

2016/0171336 A1 * 6/2016 Schwartz GO6K 9/00201 

382/173 

2016/0171707 A1 * 6/2016 Schwartz G06K 9/342 

382/180 

* cited by examiner 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 3 of 19 PageID #: 23
U.S. Patent Aug. 10 , 2021 Sheet 1 of 7 US 11,087,272 B2 

100 

102 

130 

132 133 

134 

135 

137 ? 

138 

148 

120 

122 

147 123 

125 

146 triitti 

124 

145 140 

126 

144 

LIITIT 1.11 DITITI 

143 

110 

142 

150 

141 

139 

138 

114 o Fig . 1 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 4 of 19 PageID #: 24
U.S. Patent Aug. 10 , 2021 Sheet 2 of 7 US 11,087,272 B2 

200 

O 

230 

une 

232 

205 202 127 

Fig . 2 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 5 of 19 PageID #: 25
U.S. Patent Aug. 10 , 2021 Sheet 3 of 7 US 11,087,272 B2 

300 

Robot Navigation 

and Sensing 

310 Inventory 

Cameras 

340 Inventory Camera 

Control and Image 

Reconstruction 

312 Lights 

350 

Inventory Data and 

Local Update 

314 

Communication 

System 

316 

Remote Server 

318 

Fig . 3 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 6 of 19 PageID #: 26
U.S. Patent Aug. 10 , 2021 Sheet 4 of 7 US 11,087,272 B2 

400 

Move robot along an 

aisle path 

410 

Capture and stitch 

multiple images 

together , along with 

depth information 

412 

Create panorama or 

realogram 

414 

Communicate 

panorama or 

realogram to cloud 

or remote server 

416 

Receive information 

to update product 

information 

418 Fig . 4 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 7 of 19 PageID #: 27
U.S. Patent Aug. 10 , 2021 Sheet 5 of 7 US 11,087,272 B2 

500 

510 

520 

* 

530 

0 

540 

1 

550 ******** 2.X.X materiale 

560 

Fig . 5B Fig . 5A 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 8 of 19 PageID #: 28
U.S. Patent Aug. 10 , 2021 Sheet 6 of 7 US 11,087,272 B2 

600 

Detect shelf labels 

610 

Compare shelf labels to 

depth map ( optional ) 

612 

Define a bounding box 

614 

Associate image within 

bounding box with shelf 

label 

616 

Build training data set 

( optionally , build Library of 

Products ) 

618 

Fig . 6 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 9 of 19 PageID #: 29
U.S. Patent Aug. 10 , 2021 Sheet 7 of 7 US 11,087,272 B2 

700 

Extract descriptors 

710 

Store each identifier and 

set of product descriptors 

714 

Index identifiers and 

product descriptors 

716 

Prune product objects 

718 

Fig . 7 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 10 of 19 PageID #: 30
US 11,087,272 B2 

1 2 

SYSTEM AND METHOD FOR LOCATING , product bounding box , associating the bounding box to a 

IDENTIFYING AND COUNTING ITEMS shelf label to build a training data set ; and using the training data set to train a product classifier . CROSS - REFERENCE TO RELATED PATENT In other embodiments , a system for building a product 

APPLICATION 5 library can include an image capture unit operated to provide images of items . The system also includes a shelf label 

The present disclosure is part of a non - provisional patent detector ( which can be a high resolution zoomable camera ) 

application claiming the priority benefit of U.S. Patent and optionally depth map creation unit ( which can be 

Application No. 62 / 314,785 , “ System and Method for provided by laser scanning , time - of - flight range sensing , or 

Locating , Identifying and Counting Products on Shelves ” , 10 stereo imaging ) , a processing module to optionally compare filed Mar. 29 , 2016 ; and U.S. Patent Application No. 62/427 , detected shelf labels to a depth map , define a product 

509 , " System and Method for Locating , Identifying and bounding box , and associate the bounding box with a shelf 

Counting Items ” , filed Nov. 29 , 2016 . label to build a training data set or learn image descriptors . Both the image capture unit and processing module can be 

TECHNICAL FIELD 15 mounted on an autonomous robot . Because it represents reality on the shelf , an inventory 

The present disclosure relates generally to a multiple map such as disclosed herein can be known as a “ realogram ” 

camera sensor suite capable of accurately monitoring retail to distinguish from conventional “ planograms ” that take the 

or warehouse product inventory without needing an initial form of 3D models , cartoons , diagrams or lists that show 

planogram . In certain embodiments , the multiple camera 20 how and where specific retail products and signage should 

sensor suite can be mounted on an autonomous robot and be placed on shelves or displays . Realograms can be locally include onboard processing to provide near real time product stored with a data storage module connected to the process tracking . ing module . A communication module can be connected to 

the processing module to transfer realogram data to remote 

BACKGROUND 25 locations , including store servers or other supported camera 

systems , and additionally receive inventory information 

Retail stores or warehouses can have thousands of distinct including planograms to aid in realogram construction . In 

products that are often sold , removed , added , or reposi- addition to realogram mapping , this system can be used to tioned . Even with frequent restocking schedules , products detect out of stock products , estimate depleted products , assumed to be in stock may be out of stock , decreasing both 30 estimate amount of products including in stacked piles , sales and customer satisfaction . Point of sales data can be estimate products heights , lengths and widths , build 3D 

used to roughly estimate product availability , but does not models of products , determine products ' positions and ori 

help with identifying misplaced , stolen , or damaged prod entations , determine whether one or more products are in 

ucts , all of which can reduce product availability . However , disorganized on - shelf presentation that requires corrective manually monitoring product inventory and tracking prod- 35 action such as facing or zoning operations , estimate fresh 

uct position is expensive and time consuming . ness of products such as produce , estimate quality of prod One solution for tracking product inventory relies on ucts including packaging integrity , locate products , includ 

planograms ( lists or diagrams that show how and where ing at home locations , secondary locations , top stock , specific products should be placed on shelves or displays ) in bottom stock , and in the backroom , detect a misplaced combination with machine vision technology . Given a 40 product event ( also known as a plug ) , identify misplaced planogram , machine vision can be used to assist in shelf products , estimate or count the number of product facings , space compliance . For example , large numbers of fixed compare the number of product facings to the planogram , position cameras can be used throughout a store to monitor locate labels , determine sequence of labels , detect label type , aisles , with large gaps in shelf space being checkable against read label content , including product name , barcode , UPC the planogram or shelf labels , and flagged as " out of stock ” 45 code and pricing , detect missing labels , compare label if necessary . Alternatively , a smaller number of movable locations to the planogram , compare product locations to the 

cameras can be used to scan a store aisle . Even with such planogram , determine the expiration date of products , deter 

systems , human intervention is generally required to build mine freshness of products including produce , measure shelf 

an initial planogram that includes detailed information rela- height , shelf depth , shelf width and section width , recognize tive to a bounding box that can include product identifica- 50 signage , detect promotional material , including displays , tion , placement ,and count . Substantial human intervention signage , and features and measure their bring up and down 

can also be required to update the planogram , as well as times , detect and recognize seasonal and promotional prod search for misplaced product inventory . ucts and displays such as product islands and features , capture images of individual products and groups of prod 

SUMMARY 55 ucts and fixtures such as entire aisles , shelf sections , specific products on an aisle , and product displays and islands , A low cost , accurate , and scalable camera system for capture 360 - deg and spherical views of the environment to 

product or other inventory monitoring can include a mov- be visualized in a virtual tour application allowing for virtual 

able base . Multiple cameras supported by the movable base walk throughs , capture 3D images of the environment to be 

are directable toward shelves or other systems for holding 60 viewed in augmented or virtual reality , capture environmen products or inventory . A processing module is connected to tal conditions including ambient light levels , capture infor 

the multiple cameras and able to construct from the camera mation about the environment including measuring space derived images an updateable map of product or inventory compliance with disability and safety standards and deter position . mining if light bulbs are off , provide a real - time video feed 

In some embodiments , the described camera system for 65 of the space to remote monitors , provide on - demand images inventory monitoring can be used for detecting shelf labels ; and videos of specific locations , including in live or sched optionally comparing shelf labels to a depth map ; defining a uled settings , and build a library of product images . 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 11 of 19 PageID #: 31
US 11,087,272 B2 

3 4 

In one embodiment , the movable base can be a manually The movable base 110 can be an autonomous robot 

pushed or guidable cart . Alternatively , the movable base can having a navigation and object sensing suite 130 that is 

be a tele - operated robot , or in preferred embodiments , an capable of independently navigating and moving throughout 

autonomous robot capable of guiding itself through a store a building . The autonomous robot has multiple cameras 140 

or warehouse . Depending on size of the store or warehouse , 5 attached to movable base 110 by a vertically extending multiple autonomous robots can be used . Aisles can be camera support 140. Lights 150 are positioned to direct light regularly inspected to identify out of stocks or create realo- toward target 102. The object sensing suite includes forward 

grams , with aisles having high product movement being ( 133 ) , side ( 134 and 135 ) , top ( 132 ) and / or rear ( not shown ) inspected more often . image and depth sensors to aid in object detection , local 

In another embodiment , an inventory monitoring method 10 ization , and navigation . Additional sensors such as laser includes the steps of allowing an autonomous robot to move ranging units 136 and 138 ( and respective laser scanning along an aisle that is lined with shelves capable of holding beams 137 and 139 ) also form a part of the sensor suite that 

inventory or products , with the autonomous robot acting as is useful for accurate distance determination . In certain 

a movable base for multiple cameras . Multiple cameras are embodiments , image sensors can be depth sensors that infer 

directed toward inventory on the shelf lined aisle , with data 15 depth from stereo images , project an infrared mesh overlay derived at least in part from these cameras being used to that allows rough determination of object distance in an 

construct a realogram of inventory or a panoramic image image , or that infer depth from the time of flight of light using a processing module contained in the autonomous reflecting off the target . In other embodiments , simple cam robot . Realogram data or panoramic images created by the eras and various image processing algorithms for identifying processing module can be transferred to remote locations 20 object position and location can be used . For selected 

using a communication module , and inventory information applications , ultrasonic sensors , radar systems , magnetom 

received via the communication module can be used to aid eters or the like can be used to aid in navigation . In still other 

in realogram construction . embodiments , sensors capable of detecting electromagnetic , In yet another embodiment , an inventory monitoring light , or other location beacons can be useful for precise method , includes the steps of allowing an autonomous robot 25 positioning of the autonomous robot . to move along a shelf lined aisle holding inventory , with the As seen in FIG . 1 , various representative camera types 

autonomous robot acting as a movable base for multiple useful for constructing an updatable realogram are shown . cameras . The autonomous robot can maintain a substantially As previously noted , a realogram can use camera derived 

constant or tightly controlled distance from the shelf lined images to produce an updateable map of product or inven aisle holding inventory while moving in a forward or reverse 30 tory position . Typically , one or more shelf units ( e.g. target direction . Using the multiple cameras directed toward inven 102 ) would be imaged by a diverse set of camera types , tory on the shelf lined aisle , at least part of a realogram of including downwardly ( 142 and 144 ) or upwardly ( 143 and 

inventory positioned along a shelf lined aisle holding inven- 148 ) fixed focal length cameras that cover a defined field less 

tory can be constructed . Typically , the realogram is created than the whole of a target shelf unit ; a variable focus camera 

and updated with a locally sited data storage and a process- 35 that adapts its focus to the distance from the imaged target ; 

ing module contained in the autonomous robot . To ensure a wide field camera 145 to provide greater photographic complete or near complete camera coverage of shelf lined coverage than the fixed focal length cameras ; and a narrow 

aisles , the autonomous robot can pause , reverse , or mark for field , zoomable telephoto 146 to capture bar codes , product 

further multiple camera inspection if data capture for a identification numbers , and shelf labels . Alternatively , a high 

portion of the shelf lined aisle is incomplete . 40 resolution , tilt controllable , height adjustable camera can be 

used to identify shelf labels . These camera 140 derived 

BRIEF DESCRIPTION OF THE DRAWINGS images can be stitched together , with products in the images identified , and position determined . FIG . 1 is an illustration of a camera system mounted on To simplify image processing and provide accurate 

a movable base to track product changes in aisle shelves or 45 results , the multiple cameras are typically positioned a set 

other suitable targets ; distance from the shelves during the inspection process . The 

FIG . 2 is a cartoon illustrating two autonomous robots shelves can be illuminated with LED or other directable 

inspecting opposite shelves in an aisle ; lights 150 positioned on or near the cameras . The multiple FIG . 3 is an illustration of various systems and electronic cameras can be linearly mounted in vertical ,horizontal , or 

modules connected to inventory cameras ; 50 other suitable orientation on a camera support . In some 

FIG . 4 is an illustration of steps in one embodiment of embodiments , to reduce costs , multiple cameras are fixedly operation ; mounted on a camera support . Such cameras can be arranged 

FIGS . 5A and B are respectively examples in side view to point upward , downward , or level with respect to the 

and cross section of an autonomous robot capable of acting camera support and the shelves . This advantageously per 

as a mobile base for a camera system ; and 55 mits a reduction in glare from products having highly 

FIG . 6 is a flowchart 600 illustrating in more detail how reflective surfaces , since multiple cameras pointed in 

a product space for a realogram can be created without slightly different directions can result in at least one image requiring an initial planogram ; and with little or no glare . FIG . 7 is a flow chart 700 illustrating additional process Electronic control unit 120 contains an autonomous robot 

details for managing aspects of the disclosed system . 60 sensing and navigation control module 124 that manages robot responses . Robot position localization may utilize 

DETAILED DESCRIPTION external markers and fiducials , or rely solely on localization 

information provided by robot - mounted sensors . Sensors for FIG . 1 is an illustration of an inventory monitoring position determination include previously noted imaging , camera system 100 mounted on a movable base 110 ( with 65 optical , ultrasonic sonar , radar , Lidar , Time of Flight ,struc 

drive wheels 114 ) to track product changes in aisle shelves tured light , or other means of measuring distance between 

or other targets 102 without requiring an initial planogram . the robot and the environment , or incremental distance 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 12 of 19 PageID #: 32
US 11,087,272 B2 

5 6 

traveled by the mobile base , using techniques that include server local to the store , or transmitted by suitable internet 

but are not limited to triangulation , visual flow , visual or networking devices to remote servers or cloud accessible 

odometry and wheel odometry . data sites . 

Electronic control unit 120 also provides image process- Inventory cameras 340 can include one or more movable 

ing using a camera control and data processing module 122.5 cameras , zoom cameras , focusable cameras , wide - field cam 

Autonomous robot sensing and navigation control module eras , infrared cameras , ultraviolet cameras , or other spe 124 manages robot responses , and communication module cialty cameras to aid in product identification or image 126 manages data input and output . The camera control and construction . For example , a wide - field camera can be used 

data processing module 122 can include a separate data to create an image organizing template into which data from 

storage module 123 ( e.g. solid state hard drives ) connected 10 higher resolution cameras with a narrow field of view are 

to a processing module 125. The communication module mapped or registered . As another example , a tilt control 

126 is connected to the processing module 125 to transfer lable , high resolution camera positioned on the camera 

realogram data or panoramic images to remote locations , support roughly at a height of a shelf lip can be used to read 

including store servers or other supported camera systems , shelf attached bar codes , identifying numbers , or labels . In 

and additionally receive inventory information to aid in 15 certain embodiments , conventional RGB CMOS or CCD 

realogram construction . In certain embodiments , realogram sensors can be used , alone or in combination with spectral data is primarily stored and images are processed within the filters that may include narrowband , wideband , or polariza 

autonomous robot . Advantageously , this reduces data trans- tion filters . Embodiments can also include sensors capable fer requirements , and permits operation even when local or of detecting infrared , ultraviolet , or other wavelengths to 

cloud servers are not available . 20 allow for hyperspectral image processing . This can allow , FIG . 2 is a cartoon 200 illustrating two autonomous robots for example , monitoring and tracking of markers , labels or 

230 and 232 , similar to that discussed with respect to FIG . guides that are not visible to people , or using flashing light 1 , inspecting opposite shelves 202 in an aisle . As shown each in the invisible spectrum that do not induce discomfort of 

robot follows path 205 along the length of an aisle , with health risk while reducing energy consumption and motion 

multiple cameras capturing images of the shelves 202 . 25 blur . In some embodiments , the robots 230 and 232 support at Lights can be mounted along with , or separately from , the 

least one range finding sensor to measure distance between sensors , and can include monochromatic or near monochro 

the multiple cameras and the shelves and products on matic light sources such as lasers , light emitting diodes 

shelves , with an accuracy of less than 5 cm , and with a ( LEDs ) , or organic light emitting diodes ( OLEDs ) . Broad typical accuracy range between about 5 cm and 1 mm . As 30 band light sources may be provided by multiple LEDs of 

will be appreciated , LIDAR or other range sensing instru- varying wavelength ( including infrared or ultraviolet LEDs ) , ments with similar accuracy can also be used in selected halogen lamps or other suitable conventional light source . applications . Using absolute location sensors , relative dis- Various spectral filters that may include narrowband , wide 

tance measurements to the shelves , triangulation to a known band , or polarization filters and light shields , lenses , mirrors , landmark , conventional simultaneous localization and map- 35 reflective surfaces , diffusers , concentrators , or other optics ping ( SLAM ) methodologies , or relying on beacons posi- can provide wide light beams for area illumination or tightly tioned at known locations in a blueprint or a previously built focused beams for improved local illumination intensity . map , the robots 230 and 232 can move along a path According to some embodiments , both cameras 340 and 

generally parallel to shelves 202. As the robots move , lights 350 can be movably mounted . For example , hinged , vertically positioned cameras are synchronized to simulta- 40 rail , electromagnetic piston , or other suitable actuating neously capture images of the shelves 202. In certain mechanisms used to programmatically rotate , elevate , embodiments , a depth map of the shelves and products is depress , oscillate , or laterally or vertically reposition cam 

created by measuring distances from the shelf cameras to the eras or lights . shelves and products over the length of the shelving unit In still other embodiments , one or more of the cameras using image depth sensors and or laser ranging instrumen- 45 can be mounted in such a way as to take advantage of the 

tation . The depth map is registered onto the images captured rolling shutter effects and direction of travel of the autono 

by the shelf cameras , so as the location of each pixel on mous robot . Aligning a camera in such a way as to take 

target can be estimated in 3D . Using available information , advantage of the “ rasterized ” delay of the rolling shutter can consecutive images can be stitched together to create pan- reduce artifacts ( elongation / shortening ) that can occur while oramic images that spans an entire shelving unit . The 50 the robot is traveling in its path . consecutive images can be first stitched vertically among all Inventory data 314 can include but is not limited to an 

the cameras , and then horizontally and incrementally inventory database capable of storing data on a plurality of 

stitched with each new consecutive set of vertical images as products , each product associated with a product type , the robots 230 and 232 move along an aisle . product dimensions , a product 3D model , a product image FIG . 3 is an illustration of various systems and electronic 55 and a current product price , shelf location , shelf inventory modules 300 supported by an autonomous robot having count and number of facings . Realograms captured and 

robot navigation and sensing 310. Inventory cameras 340 created at different times can be stored , and data analysis 

are moved into a desired position with the aid of robot used to improve estimates of product availability . In certain navigation and sensing module 310. Lights 350 are directed embodiments , frequency of realogram creation can be 

toward product inventory and inventory camera control and 60 increased or reduced . image reconstruction 312 takes a series of inventory photos The communication system 316 can include connections 

( and optional depth measurements ) that can be stitched to either a wired or wireless connect subsystem for interac 

together to help form or update a realogram . Panoramic tion with devices such as servers , desktop computers , lap images , realogram data , or other inventory related informa- tops , tablets , or smart phones . Data and control signals can 

tion is handled by an inventory data and local update module 65 be received , generated , or transported between varieties of 

314 , which can transmit or receive relevant information via external data sources , including wireless networks , personal communication system 316. Data can be communicated to a area networks , cellular networks , the Internet , or cloud 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 13 of 19 PageID #: 33
US 11,087,272 B2 

7 8 

mediated data sources . In addition , sources of local data ( e.g. Inventory monitoring can rely on use of autonomous 

a hard drive , solid state drive , flash memory , or any other robot camera system images . Typically , multiple images are 

suitable memory , including dynamic memory , such as processed , combined , and segmented for further analysis . SRAM or DRAM ) that can allow for local data storage of Segmented images can assist in defining a product bounding 

user - specified preferences or protocols . In one particular 5 box that putatively identifies a product facing . This infor 

embodiment , multiple communication systems can be pro mation is often necessary to develop a product library . A 

vided . For example , a direct Wi - Fi connection ( 802.11b / g / n ) segmented image can include multiple product bounding 

can be used as well as a separate 4G cellular connection . boxes , typically ranging from dozens to hundreds of out 

Remote server 318 can include , but is not limited to lined or distinct image areas . The bounding boxes can 10 surround either product facings , groups of products , or gaps servers , desktop computers , laptops , tablets , or smart 

phones . Remote server embodiments may also be imple between products . Products within product bounding boxes 

can be manually identified , identified using crowd source or mented in cloud computing environments . Cloud computing paid reviewer image identification systems , identified with may be defined as a model for enabling ubiquitous , conve or without the aid of an initial planogram , or automatically nient , on - demand network access to a shared pool of con 15 identified using various image classifiers discussed herein . figurable computing resources ( e.g. , networks , servers , stor Gaps between products are useful for identifying shelf 

age , applications , and services ) that can be rapidly spacings , product separation , or missing / absent inventory . provisioned via virtualization and released with minimal Automatic identification can be performed using an 

management effort or service provider interaction , and then autonomous robot , alone or in combination with an external scaled accordingly . A cloud model can be composed of 20 image classifier system . In certain embodiments , a product various characteristics ( e.g. , on - demand self - service , broad bounding box can be defined as the horizontal space on the 

network access , resource pooling , rapid elasticity , measured shelf occupied by one or more copies ( facings ) of the same 

service , etc. ) , service models ( e.g. , Software as a Service product , along with the vertical space spanning the distance 

( “ SaaS ” ) , Platform as a Service ( “ PaaS ” ) , Infrastructure as a between a current shelf and the shelf above it . When the 

Service ( “ IaaS ” ) , and deployment models ( e.g. , private 25 current shelf is the top shelf , the vertical space is a number cloud , community cloud , public cloud , hybrid cloud , etc. ) . generally corresponding to the distance to top of the fixture . FIG . 4 is an illustration of realogram or panorama updat- The vertical space can alternatively be top of the product as 

ing steps in one embodiment of operation . As seen in flow sensed by depth sensors . chart 400 , a robot moves to an identified position and Image segmentation to automatically assist in creation of 

proceeds along an aisle path at a predetermined distance 30 product bounding boxes and product identification can rely ( step 410 ) . If the path is blocked by people or objects , the on use of image templates in some embodiments . Typically , robot can wait till the path is unobstructed , begin movement each image template is compared with the image captured and slow wn , wait as it nears the obstruction , move along by a camera system mounted on an autonomous robot . If a 

the path until required to divert around the object before match is positive , the matched section of the image is used reacquiring the path , or simply select an alternative aisle . 35 as the image segmentation for that product In step 412 , multiple images are captured and stitched Segmentation can be improved by training classifiers on 

together to define an image panorama . Optionally , in certain annotated training data sets , where bounding boxes are embodiments a panoramic or widefield camera can capture manually drawn around products . Training can be performed 

a single large image . These images , along with optional with supervised or unsupervised machine learning , deep depth information created by a laser ranging system , an 40 learning , or hybrid machine and deep learning techniques , infrared depth sensor , or similar system capable of distin- including but not limited to convolutional neural networks . guishing depth at a decimeter or less scale , are used to create Some methods include reducing the number of image either panorama or a realogram ( step 414 ) . This information templates that must be considered by only matching the 

is communicated to a cloud or remote server ( step 416 ) to templates that correspond to the product identifier ; or to the 

create , change , or update a panorama and / or realogram with 45 product objects who are proximal to the shelf location being data derived from shelf labels , bar codes , and product scanned . Product objects can include but are not limited to : identification databases to identify products . A realogram is a product identifier 

created using panorama images and data , and can be used by , One or more sets of descriptors for example , store managers , stocking employees , or cus- Confidence levels for each set of descriptors 

tomer assistant representatives to localize product and label 50 One or more shelf position metric estimates 

placement , estimate product count , count the number of Confidence levels for each shelf position metric estimate 

product facings , or even identify or locate missing products . One or more shelf position topological estimates 

Additionally , in some embodiments , realogram or other Count number for each shelf position topological estimate 

information received from other robots , from updated prod- One or more image templates of the product 

uct databases , or from other stores can be used to update or 55 Dimensions of the product assist in the creation of subsequent realograms ( step 418 ) . Product objects can be updated , manually or automati 

FIGS . 5A and B are respectively examples in side view cally revised , augmented , or corrected , and changed to 

and cross section of an autonomous robot 500 capable of match changing product specifications . acting as a mobile base for a camera system in accordance Some methods further detect if a product is oriented with this disclosure . The robot navigation and sensing unit 60 differently than the externally sourced image . If a template includes a top mount sensor module 510 with a number of match cannot be found , but the product descriptor does find 

forward , side , rear , and top mounted cameras . A vertically a high - likelihood match , this is indicative of a product aligned array of lights 520 is sited next to a vertically orientation that is different from that of the image sourced 

arranged line of cameras 530 , and both are supported by a externally to the mobile base system . If the externally drive base 540 that includes control electronics , power , and 65 sourced image is known to be a front view of the product , docking interconnects . Mobility is provided by drive wheels then this method identifies products that are improperly 560 , and stability is improved by caster wheels 550 . oriented on the shelf . The angular deviation of the improp 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 14 of 19 PageID #: 34
US 11,087,272 B2 

9 10 

erly oriented product can be estimated and an affine trans- Depth camera using triangulation formation between the set of descriptors of the externally Planar or 3D laser / lidar scanner 

sourced image and the segmented portion of the image Color segmentation ; 

computed . Product features including but not limited to product For situations where template matching is successful , the 5 shapes , colors , texts , and aspect ratios ; 

product segmentation in the image can be considered accu- Product shapes identified and learned using machine 

rate , and the real dimensions of the product are compared to learning techniques such as convolutional neural networks 

the apparent dimensions of the product in the image to and deep learning 

extract a distance estimate between the imaging sensor and Association of individual product images to identifiers the product . Additionally , the apparent position of the prod- 10 based on a location heuristic . The heuristic may locate the 

uct in the image combined with the distance estimate enable identifier below a product image or in another proximal the computation of the three - dimensional position and ori- location . entation between the imaging sensor and the product . The heuristic may be informed from the planogram by Some methods extract topological shelf positions for each correlating the planned location of labels to the measured 

product from the planogram . These methods further increase 15 location of labels , and the number of facings for each facing the scope of the product object identified through segmen- group with the measured distance between consecutive 

tation and redefine it as containing : labels divided by the width of the product corresponding to 

One product identifier the left label . These correlations can be optimized , for 

One or more sets of descriptors example by using graph theory approaches , to generate a 

Confidence levels for each set of descriptors 20 maximum likelihood correspondence between a facing One or more shelf position metric estimates group and a label . Confidence levels for each shelf position metric estimate Association may also be informed by mapping the left 

One or more shelf position topological estimates most label on a shelf with the left most group of self - similar 

Count number for each shelf position topological estimate facings , the right most label on the same shelf with the right One or more image templates of the product 25 most label facing group , and working inward until every Dimensions of the product facing group has an associated label . One or more topological shelf positions from planogram Further , association may be informed by classifiers 

In other embodiments , RFID tags , wireless beacons , trained on hand - annotated associations from training data 

locators , or trackers can be used alone or in combination to sets and using heuristics similar to the ones described above . assist in defining a product bounding box . For example , in 30 Some methods further include inferring the position of each 

some embodiments , an autonomous robot can be addition- identifier on the shelf by comparing the location of the 

ally equipped with one or more RFID readers . Performing an identifier to the location of the beginning or end of the shelf inventory count of products equipped with RFID tags can on which the identifier is placed . Alternatively , methods can 

proceed in one embodiment as follows : be based on inferring the position of each product on the 

The total number of tags for each product is communi- 35 shelf by comparing the location of the product to the location 

cated by an inventory management software to an RFID of the beginning or end of the shelf on which the product is 

reader mounted on or associated with an autonomous robot ; placed The RFID reader collects RFID tags while the autono- In some methods , an identifier's shelf location and / or 

mous robot is either stationary or moving ; product location are expressed in metric terms ,i.e. measured 

If the RFID reader does not collect all the tags for a given 40 distance from the beginning or end of a specific shelf . In 

product , and : other methods , an identifier's shelf location and / or product If the autonomous robot is moving , then the autonomous location are expressed topologically , e.g. as a sequence of 

robot stops in an attempt to collect the remaining tags , or identifiers from the start or the end of a specific shelf , and 

If the autonomous robot is stopped , move the autonomous from the bottom of a shelf to the top , or from top to bottom . robot in a predetermined search path in an attempt to collect 45 For example , a specific identifier may be third from the start 

the remaining tags . of the fourth shelf . With suitable changes , Bluetooth , Near Field Communi- If a product library is created or made available , the 

cation , or other conventional wireless system can be used in library can be searched for products objects with a large place of RFID systems . number of similar features to assist in developing a product In some embodiments , visual images based on numeric , 50 bounding box . For each potential product object match , the 

alphabetic , one or two - dimensional bar codes , or similar geometric consistency of the feature locations in the library image based shelf or product labels can be used alone or in can be compared with the features in a shelf image . Some combination with various image features to segment images methods further include indexing the sets of descriptor and assist in defining a product bounding box . Each shelf within the library for improved searching performance and / image can be analyzed and segmented to detect individual 55 or reduced storage requirements . Indexing methods include price tags or product labels on the shelf . Alternatively , or in but are not limited to : hashing techniques , tree representa addition , each image can be used to detect identifying tions , and bag - of - words encodings . Alternatively , plano indicia of individual products and product packages . Seg- gram information or product location information from the 

mentation can use techniques that include but are not limited product library can be used to reduce the number of products 

to : 60 that must be searched to just those products contained within Edge detection ; the imaged shelf . In still other variations , identified products Depth estimation using depth estimation techniques that can be verified by segmenting and decoding the price tag or 

include but are not limited to : product label located proximally to each identified product 

Stereo camera and comparing it to the product object identifier . Structure from motion FIG . 6 is a flowchart 600 illustrating in more detail one 

Structure from focus example of how a product bounding box such as previously Depth camera using time of flight discussed can be created from information captured by 

65 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 15 of 19 PageID #: 35
US 11,087,272 B2 

11 12 

sensor and camera system supported by an autonomous For vertically stacked products , the height of the bound 

robot . Turning to FIG . 6 , in a first step 610 , shelf labels are ing box is subdivided in sections equal to the height of the 

detected either in individual shelf images , or in stitched product . panorama . Classification algorithms such as convolution In certain embodiments , the previous method for deter neural networks or other deep learning methods , template 5 mining the bounding box can be probabilistically combined 

matching or HAAR cascades can be used to aid in detection through estimating a confidence measure for each method 

of each shelf label . Each shelf label is analyzed to obtain one and adding their results . 

or more product identifiers . Analysis may include but is not Next , in step 616 , each bounding box is consistently associated with an identifier based on a label location limited to optical character recognition , bar code scanning , QR code scanning , AR code scanning , or hologram code 10 heuristic . The identifier can be selected to originate from either the left or the right shelf label . The association of scanning . Product identifiers may be UPC code , the product bounding boxes and identifier can further be refined through name , or a coded collection of letters , numbers , or other optimization across an entire shelf section or aisle . The symbols . If more than one identifier is available , a preferred bounding box , with identifiers , can be registered to a simple identifier such as the UPC code can be selected . In certain 15 or panoramic stitched image of the shelf , and image descrip embodiments , infrared or ultraviolet detectable product tors extracted for the portion of the image contained in the identifiers embedded on product packaging or shelf labels bounding box . Methods for generating image descriptors 

can be used , as well as any other suitable tag , marker , or include but are not limited to : image templates , Histogram detectable identifying indicia such as a visible UPC code or of Gradients , Histogram of Colors , the Scale Invariant serial number on the product packaging . 20 Feature Transform , Binary Robust Independent Elementary In optional step 612 , an image location of the shelf label Features , Maximally Stable Extremal Regions , Binary is registered or compared to a depth map to recover its 3D Robust Invariant Scalable Keypoints , Fast Retina Key location in space . The depth map can be created by use of points , Kaze features , and variations thereof . one or more depth sensors that infer depth from stereo An alternative to extracting product descriptors is to use 

images , project an infrared mesh overlay that allows rough 25 the bounding boxes as labeled categories and train classifiers 

determination of object distance in an image , infer depth on the images contained in the bounding boxes . Classifiers 

from the time of flight of scanning lasers or LEDs reflecting may include those based on deep structured learning , hier 

off the target , or any other suitable method for building a archical learning , deep machine learning , or other suitable 

depth map typically having sub - millimeter to sub - centimeter deep learning algorithms associated with convolutional , resolution . 30 feedforward , recurrent , or other suitable neural network . A 

In step 614 a bounding box is defined as a perimeter deep learning based classifier can automatically learn image 

enclosing one or multiple facings of the same product , or descriptors based on an annotated training data . For example , deep learning based image descriptors can be 

any space on the shelf including but not limited a gap 

between products . The bounding box can be manually 35 volutional neural networks . The final layer of a convolu hierarchical , corresponding to multiple layers in deep con 

defined , or automatically defined using training classifiers , tional layer network outputs the confidence values of the deep learning , image segmentation , or any other suitable product being in one of the designated image categories . The technique or combination of techniques . A bounding box can image descriptor generator part and the classification part be created with reference to labels grouped by height , with get integrated in a convolutional neural network and these 

a horizontal distance between adjacent labels used to define 40 two parts are trained together using a training set . the width of the bounding box for single facing products . For Alternatively , or in addition , embodiments that use both multiple facing products , the bounding box width is subdi- deep learning based image descriptors and conventional 

vided in sections equal to the product width . image descriptors can be combined in a hybrid system . The height of the bounding box can be derived from the In step 618 , the image descriptors can be classified and 

detection of shelf heights . Shelf heights can be detected by 45 labelled with the identifier . Classification algorithms that can 

analyzing the depth map to identify horizontal indentations include but are not limited to support vector machine . This corresponding to the distance between the bottom of a shelf process can be repeated for every image of the bounding box 

and products stacked below it . These horizontal indentations associated to the same identifier , whether the image is correspond to shelf lips and measure shelf height .captured in the same store at different times , or in different 

Alternatively , label groups can be filtered by horizontal 50 stores . In time , this allows automatically building a product 

span ( defined as the horizontal distance between the first and library ( i.e. the “ Library of Products ” ) , without requiring an 

last label of the group ) according the following criteria : initial planogram or storage of specific product databases . A label group passes the filter if its horizontal span For those embodiments utilizing deep learning based 

overlaps with the current label group span . image descriptors , the neural network classifier can be part A label group passes the filter if its horizontal span is 55 of the same trained convolutional neural network . The distant from the current label group span by no more than a automatically learned features , which are extracted from 

number representing a bay width . Generally , bay widths are different layers of the convolutional neural network , can be 

standard three or four - foot - wide shelves used throughout a used in a larger product recognition system . These features 

store . can be used in conjunction with other classifiers or with 

Order filtered label groups by height and select the label 60 conventional image descriptors in a hybrid system . group that is next highest after the current label group . FIG . 7 is a flow chart 700 illustrating additional process Alternatively , shelf heights can also be detected by deep details for managing one embodiment of a library of prod learning classifiers trained on manually annotated color and ucts . The library of products can be stored in a single depth images ( if available ) of the shelves . autonomous robot such as described herein , distributed The height of the bounding box can be fully defined once 65 between two or more autonomous robots , or stored in a 

the height difference between the selected label group and local , remote , or cloud server , in whole or in part . In one 

the current label group is determined . embodiment ,step 710 requires extraction of one set of 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 16 of 19 PageID #: 36
US 11,087,272 B2 

13 14 

descriptors from each product image associated to an iden- heuristic value . Otherwise , keep the data from last two dates 

tifier . One or more sets of descriptors can be developed , or more , until a confidence level above the heuristic value is 

confidence levels for each set of descriptors set ; dates at found . which each set of descriptors was generated recorded ; one or vi ) Topological shelf positions : delete all topological 

more shelf position metric estimates made ; confidence levels 5 positions along with their counter number and date entries , for each shelf position metric estimated ; one or more shelf except the latest date , if its associated count number is position topological estimated ; or a count number for each higher than a heuristic value . Otherwise , keep the data from 

shelf position topological estimated . In other embodiments , last two dates or more , until a count number above the 

one or more image templates of the product or dimensions heuristic value is found . of the product can be used to determine product object 10 In some embodiments , modifying the product library can 

scope . utilize external image data sources to aid in product identi 

In step 714 , each identifier and set of product descriptors fication . These images can be acquired from one or more of is stored in a database or library of product objects as the following sources : 

follows : Retailer carrying the product , where the image corre 

If the identifier does not match an existing identifier in the 15 sponds to the product identifier library , create a new object containing the identifier , the set Manufacturer of the product , after matching the product of product descriptors , an entry confidence level for the set identifier with the manufacturer's nomenclature and image of product descriptors , and the date . The entry confidence Third - party photography of the product level is unique . Online databases , image search engines , or data from 

If the product's identifier matches an existing identifier in 20 online retailers of product listings , or other Internet data 

the library , compute a score for each existing set of descrip- bases . Searches are conducted preferably using the manu 

tors that describes the likelihood that the new and existing facturer's nomenclature that corresponds to the product sets of descriptors were extracted from the same object . identifier . In some methods , this search is performed auto 

If the highest likelihood score exceeds a heuristic value matically and the one or more images inserted without 

for a positive match : combine the new set of descriptors with 25 human intervention . the highest likelihood feature set ; and increase the confi- The one or more externally derived images can include at 

dence level of this set of descriptors relative to the other sets least one front view of the product , and optionally additional 

of descriptors in the object ; and append the date to that of the views , such as back , sides , top and bottom , or different existing set of descriptors . packaging appearances of the product that correspond to the 

If the highest likelihood score does not exceed a heuristic 30 same identifier . If a product is oriented differently than the 

value for a positive match : add the set of descriptors to the externally sourced image , the product descriptor can attempt object ; set the confidence to the entry level ; and add the date . to find a high - likelihood match . In some embodiments , the 

For those embodiments utilizing deep learning based angular deviation between a likely product and a differently image recognition , the input image is classified belonging to oriented product can be determined by computing an affine 

one of the product categories using convolutional neural 35 transformation between the set of descriptors of the exter 

network which outputs the confidence level . This confidence nally sourced image and the available product image . level is then used to store the image descriptors in the library For situations where template matching is successful ,and 

in the same fashion as above . the product segmentation in the image is accurate , the real 

In step 716 , the descriptor sets within the library can be dimensions of the product can be compared to the apparent used for improved searching performance and / or reduced 40 dimensions of the product in the image to extract a distance 

storage requirements . Indexing methods include but are not estimate between the imaging sensor and the product . Addi limited to : hashing techniques , tree representations , and tionally , the apparent position of the product in the image bag - of - words encodings . combined with the distance estimate enable the computation In step 718 , the product library can optionally be pruned of the three - dimensional position and orientation between 

to reduce errors and database size . Pruning of sets of 45 the imaging sensor and the product . descriptors and metric or topological shelf positions can In the deep learning based alternative embodiments , the 

occur as follows : product segmentation can be performed by a convolutional 

Sets of descriptors : delete all sets whose confidence level neural network which classifies pixels as belonging to is below a heuristic value , along with their confidence level product interior and product background . The product and date entries , except if they originate from images 50 dimensions are inferred from the segmentation . sourced externally to the mobile base system . Many modifications and other embodiments of the inven 

Metric shelf positions : delete all metric positions whose tion will come to the mind of one skilled in the art having confidence level is below a heuristic value except if their the benefit of the teachings presented in the foregoing date of entry is the last date on record , along with their descriptions and the associated drawings . Therefore , it is 

confidence level and date entries 55 understood that the invention is not to be limited to the 

Topological shelf positions : delete all topological posi- specific embodiments disclosed , and that modifications and 

tions whose count number is below a heuristic value except embodiments are intended to be included within the scope of 

if their date of entry is the last date on record , along with the appended claims . It is also understood that other embodi 

their count number and date entries ments of this invention may be practiced in the absence of 

Alternatively , pruning can involve : 60 an element / step not specifically disclosed herein . iv ) Sets of descriptors : delete all sets except for the set 

with the highest confidence level , along with their confi- The invention claimed is : 

dence level and date entries , except if they originate from 1. A method comprising : 

images sourced externally to the mobile base system . capturing one or more images of a portion of a shelf with 

v ) Metric shelf positions : delete all metric positions along 65 a camera ; 

with their confidence level and date entries , except the latest detecting and reading content of a shelf label on the 

date , if its associated confidence level is higher than a portion of the shelf within the one or more images ; 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 17 of 19 PageID #: 37
10 

boxes ; 

15 

20 

30 

US 11,087,272 B2 

15 16 

defining one or more product facing bounding boxes 8. The method of claim 7 , wherein capturing one or more 

surrounding one or more corresponding facings of a further images comprises capturing the one or more further 

product within the one or more images ; images subsequent to capturing the one or more images . associating the shelf label content with each of the one or 9. The method of claim 7 , wherein capturing one or more 

more product facing bounding boxes that includes and 5 further images comprises capturing the one or more further 

defines each of the one or more corresponding product images prior to capturing the one or more images . facings ; 10. A method comprising : 

cross - correlating the one or more product facing bounding capturing one or more images of a portion of a shelf with 

boxes and one or more other product facing bounding a camera ; 

boxes across one or more stores by matching the shelf detecting and reading content of a shelf label on the 

label content with other shelf label content associated portion of the shelf within the one or more images ; 

with the one or more other product facing bounding defining one or more product facing bounding boxes 

surrounding one or more facings of a product within the 

building training data based on the association of the shelf one or more images ; 

label content with the cross - correlated product facing associating the shelf label content with each of the one or 

bounding boxes ; more product facing bounding boxes that includes and 

using the training data to build a product classifier ; and defines each of the one or more corresponding product using the product classifier to classify an item as the facings ; 

product . cross - correlating the one or more product facing bounding 2. The method of claim 1 , wherein capturing one or more boxes and one or more other product facing bounding images comprises capturing the one or more images with a boxes across one or more stores by matching the shelf 

camera attached to a moveable robot . label content with other shelf label content associated 

3. The method of claim 1 , further comprising using with the one or more other product facing bounding manual input to identify potential inventory designated in 25 boxes ; 

the one or more product facing bounding boxes . building training data based on the association of the shelf 

4. The method of claim 1 , wherein using the training data label content with the cross - correlated product facing 

to build a product classifier comprises integrating training bounding boxes ; 

to : ( 1 ) learn product image descriptors and ( 2 ) classify items using the training data to build a product classifier ; and 

into a neural network . using the product classifier to perform an inventory man 

5. The method of claim 1 , wherein associating the shelf agement activity related to the product . 11. The method of claim 10 , wherein capturing one or label content with each of the one or more product facing more images of a portion of a shelf comprises capturing one bounding boxes comprises manually associating the shelf or more images of a portion of a shelf in a first store ; and label content with each of the one or more product facing further comprising : bounding boxes . capturing one or more further images of a portion of 6. The method of claim 1 , wherein associating the shelf another shelf in a second store ; label content with each of the one or more product facing reading content of another shelf label on the portion of bounding boxes comprises automatically associating the the other shelf within the one or more further images ; 

shelf label content with each of the one or more product 40 defining one or more other product facing bounding facing bounding boxes . boxes surrounding one or more other corresponding 7. The method of claim 1 , wherein capturing one or more facings of another product within the one or more images of a portion of a shelf comprises capturing one or further images ; and 

more images of a portion of a shelf in a first store ; and associating the other shelf label content with each of further comprising : the one or more other product facing bounding boxes 

capturing one or more further images of portion of that includes and defines each of the one or more 

another shelf in a second store ; other corresponding product facings ; 

reading content of another shelf label on the portion of wherein cross - correlating comprises cross - correlating the the other shelf within the one or more further images ; one or more product facing bounding boxes and the one 

defining one or more other product facing bounding 50 or more other product facing bounding boxes across the 

boxes surrounding one or more other corresponding first store and the second store by matching the shelf 

facings of another product within the one or more label content with the other shelf label content ; and 

further images ; and wherein building a training data comprises building the 

associating the other shelf label content with each of training data set based on the association of the shelf 

the one or more other product facing bounding boxes 55 label content and the other shelf label content with the 

that includes and defines each of the one or more cross - correlated product facing bounding boxes corre other corresponding product facings ; lated across the first store and the second store . wherein cross - correlating comprises cross - correlating the 12. The method of claim 11 , wherein capturing one or 

one or more product facing bounding boxes and the one more further images comprises capturing the one or more 

or more other product facing bounding boxes across the 60 further subsequent to capturing the one or more images . first store and the second store by matching the shelf 13. The method of claim 11 , wherein capturing one or 

label content with the other shelf label content ; and more further images comprises capturing the one or more wherein building training data comprises building the further images prior to capturing the one or more images . training data based on the association of the shelf label 14. The method of claim 10 , wherein capturing one or 

content and the other shelf label content with the 65 more images of a portion of a shelf with a camera comprises 

cross - correlated product facing bounding boxes corre- capturing the one or more images with a camera attached to 

lated across the first store and the second store . a moveable robot . 

35 

45 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 18 of 19 PageID #: 38
10 

15 

US 11,087,272 B2 

17 18 

15. The method of claim 10 , further comprising using 19. The system of claim 18 , wherein instructions config manual input to identify potential inventory designated in ured to capture one or more images of a portion of a shelf 

the one or more bounding boxes . comprise instructions configured to capture one or more 16. The method of claim 10 , wherein the one or more images of a portion of a shelf in a first store ; and bounding boxes surround gaps between potential inventory . 5 further comprising instructions configured to : 17. The method of claim 10 , wherein associating the shelf label content with each of the one or more product facing capture one or more further images of a portion of 

bounding boxes comprises another shelf in a second store ; automatically associating the 

shelf label content with each of the one or more product read content of another shelf label on a portion of the 

facing bounding boxes . other shelf within the one or more further images ; 

18. A system comprising : define one or more other product facing bounding 

a processor ; boxes surrounding one or more other corresponding 

system memory coupled to the processor and storing facings of another product within the one or more 

instructions configured to cause the processor to : further images ; and 

associate the other shelf label content with each of the capture one or more images of a portion of a shelf with one or more other product facing bounding boxes a camera ; that includes and defines each of the one or more detect and reading content of a shelf label on the 

portion of the shelf within the one or more images ; other corresponding product facings ; 

define one or more product facing bounding boxes that wherein instructions configured to cross - correlate the one 

surround an surrounding one or more corresponding or more product facing bounding boxes and the one or 

facings of a product within the one or more images ; more other product facing bounding boxes across the 

associate the shelf label content with each of the one or first store and the second store by matching the shelf 

label content with the other shelf label content ; and more product facing bounding boxes that includes 

and defines each of the one or more corresponding wherein instructions configured to build a training data 

product facings ; comprising building the training data sent based on the 

association of the shelf label content and the other shelf cross - correlate the one or more product facing bound ing boxes and one or more other product facing label content with the cross - correlated product facing 

bounding boxes across one or more stores by match bounding boxes correlated across the first store and the 

second store . ing the shelf label content with other shelf label 

content associated with the one or more other prod- 30 ured to capture one or more further images comprise instruc 20. The system of claim 18 , wherein instructions config 

uct facing bounding boxes ; 

build training data based on the association of the shelf tions configured to capture the one or more further images 

label content with the cross - correlated product fac subsequent to capturing the one or more images . ing bounding boxes ; 21. The system of claim 18 , wherein instructions config 

use the training data to build a product classifier ; and ured to capture one or more further images of another 

use the product classifier to perform one or more of : ( 1 ) portion of a shelf comprise instructions configured to cap 

an inventory management activity related to the ture the one or more further images . product or ( 2 ) classifying an item as the product . 

20 

25 

35 

* * * * 

Case 2:23-cv-00617-RWS-RSP Document 1-2 Filed 12/19/23 Page 19 of 19 PageID #: 39