Patent ID: 12260641

DETAILED DESCRIPTION

Hereinafter, implementations of the present invention will be described with reference to the drawings. These implementations are provided for better understanding of the present invention, and the present invention is not limited only to the implementations. Changes and modifications apparent from the implementations still fall in the scope of the present invention. Meanwhile, the original claims constitute part of the detailed description of this application.

Food Preparation Station

Restaurants use food preparation stations in their kitchens. A typical food preparation station has a food preparation table and food pans containing food ingredients. Restaurant workers (workers) prepare a food on the food preparation table using ingredients from the food pans.

Recipe Guidance and Food Pan Indicating Light

To help workers prepare food, guidance for preparing food may be provided on the food preparation station. Workers may follow such instructions to prepare food. The station may be provided with indication lights for indicating food pans. To help workers locate ingredients quickly, the station may turn on an indicating light to indicate a food pan containing a particular ingredient to be used at a particular step of the instructions. Sometimes, however, the food pan indicated with the indicating light may contain another ingredient, which may confuse workers.

Tracking Changes of Ingredient Location

An enhanced food preparation station may be associated with a system that tracks location changes of the food pans or ingredients contained in the food pans. The system may have the very current location of each ingredient contained in each food pan. Then, the system can use the accurate location of each ingredient from the system and turn on the indicating light(s) for indicating the correct ingredient to be used at each step of the instructions. The configuration and operation of an enhanced food preparation station will be described with reference to an example recipe.

Pepperoni Pizza

FIG.1illustrates a flow chart for preparing a pepperoni pizza on a food preparation station before the pizza is baked in a pizza oven or furnace. Step 1 is preparing a dough, which is followed by Step 2 for adding sauce on the dough. Then, at Step 3, cheese is added over the sauce, which is followed by Step 4 for adding pepperoni over cheese. As exemplified, a flow of preparing a pizza includes steps of sequentially stacking a food ingredient over a pizza dough. While a pepperoni pizza recipe is discussed herein, the station can guide a person to prepare different pizzas and various dishes other than pizzas.

Food Preparation System

Food Preparation Station

FIG.2Aillustrates a kitchen system according to an implementation.FIG.2Billustrates a side view of the station ofFIG.2A.FIG.3illustrates a food pan array viewed from the top. The food preparation station100ofFIG.2Aincludes a food preparation table110and a food pan array120. The station100further includes a display130, light indicators140, at least one camera150, a computing system160, a database170, and an ID card reader180.FIG.4AtoFIG.4Dare photographs of an example food preparation station4100.

Food Preparation Table

The food preparation table110provides a working surface on which food is prepared.FIG.2Bshows a person210preparing a pizza220on the table110. The table110is adjacent to the food pan array120such that the person210can pick up food ingredients from the array120without having to step toward the array120. The station ofFIG.4Ahas a food preparation table4120with two pizzas4121,4122being prepared. The table4120is sized such that two persons can work at the same time.

Food Pan Array

A food pan array is for temporarily storing food ingredients. The food pan array120ofFIG.3includes a frame310and a plurality of food pans320placed on the frame310.FIG.4Bshows another food pan array4110. In the example ofFIG.3, the food pans320are arranged in 6 columns and 2 rows. A food pan array may have a different arrangement from the examples.

Food Pans

In an implementation, each one of the food pans320is a container for storing one or more food ingredients. The pans may be in the same size or different sizes. The pans may be in the same shape or different shapes. A food pan may be used with or without a lid or cover.FIG.4Bexample food pans4420containing ingredients to prepare pizzas.

Food Pan Frame—Rail Structure

In an implementation, the frame may have a rail structure on which one or more food pans are placed. Referring toFIG.4B, the food pan array4110have two elongated bars (rails)4410on which food pans4420are placed in a row. Each food pan has a flange to be slidably placed on the two elongated rails such that each food pan can slide along the rails4410and change its location in the array4110.

Food Pan Frame—Recesses

In an implementation, the frame may include a plurality of recesses (or holes), each of which is to receive one or more food pans. One or more food pans can be placed into each recess. In embodiments, a frame may have a structure different from the examples for holding one or more food pans.

Light Indicators

In an implementation, light indicators are used to visually indicate locations of food ingredients. Referring toFIG.3, a light indicator141is provided above a pepperoni pan321. When pepperoni is needed for the pizza220, the indicator141may be selectively turned on to draw the person's attention to the pan321and to indicate location of pepperoni while the other light indicators are not turned on. Alternatively, to indicate the pepperoni pan321, the indicator141may be turned off while all the other light indicators are turned on.

Location of Light Indicators

InFIG.2A, for example, the light indicators140are installed on the frame310. In implementations, one or more lights may be attached to a pan of the array120such that the lights are visible to the person210. In implementations, a lighting device such as a spotlight installed over the station may highlight a particular food pan to indicate ingredient contained therein.

Positional Association Between Indicator and Pan

Light indicators may be arranged according to a predetermined layout from which the person210can recognize which pan is associated which light and will pay attention to a particular pan when an indicator is on. For example, inFIG.3, a series of light indicators142are installed along an upper edge of the frame310and above Row2of food pans. The light indicators142are sized and arranged such that each indicator is positioned right above its corresponding food pan of Row2. From the arrangement, the person210recognizes that the indicator141is associated with the pepperoni pan321as it is the closest to the pan321, and will pay attention to the pepperoni pan321when the indicator141is on. InFIG.3, for another example, a light strip144is installed along a lower edge of the frame310and under Row1of food pans, and a group of six lights146is right under the sauce pan323. Turning on the six lights146would suggest the person210to pay attention to the sauce pan323rather than other pans because the sauce pan323is the closest pan right above the lights146.

Indicator not Suggesting a Particular Pan

InFIG.3, among the lights145of the light strip144, two lights148are not distinctively close to a particular pan, and do not overlap any food pan along a column direction. While the system may turn on a group of lights147to indicate the cheese pan324and turn on another group146to indicate the sauce pan324, the system may not turn on the two lights148interposed between the two groups146,147. In implementations, the system may not operate an indicator in association with a particular food pan when the person would not recognize that the pan is associated with the indicator from the indicator's location on the frame310.

Two or More Indicators for a Single Pan

In implementations, two or more indicators are assigned to a single food pan. Referring toFIG.4D, a light indicator4140includes two LED light strips4141,4142installed above a food pan4421. The two strips4141,4142may operate together or independently to draw a person's attention to the pan4421. When two pizzas4121,4122are being prepared on the table4120as shown inFIG.4B, the lower strip4141may be turned when the pan's ingredient is needed for the left pizza4121, and the upper strip4142may be turned on when the pan's ingredient is needed for the right pizza4122although not limited thereto.

Controlling Indicators Referencing to Database

To indicate locations of food ingredients using light indicators, the system may have location information for each indicator and also have information of which indicator is associated which ingredient. In implementations, for each food ingredient, the system stores the location of the ingredient in connection with one or more light indicators that has positional association with the ingredient as exemplified inFIG.8. When an ingredient is needed to prepare the pizza220, the system may locate one or more light indicators to turn on based on link between the ingredient and the one or more light indicators on the database.

Operation Modes of Light Indicators

A light indicator may stay turned-on, flashes, or change its color and brightness to indicate location of its corresponding food ingredient or to indicate a status of the food ingredient. The light indicator may operate in a way different from the example to draw the person's attention.

Display

The display130is for displaying food preparation information for the person210working at the station100. For example, the display130may display one or more of a received order, instructions to prepare an ordered pizza, the current progress of pizza preparation, and a performance feedback after the pizza is prepared.

Location of Display

The display130may be placed over the food pan array100although not limited thereto. In an implementation, the display130may be installed next to the table such that the person can see the pizza200and the display130at the same time. In implementations, the display130is facing the person210such that the person can read information on the display while preparing the pizza220on the table110.

Two or More Displays

In an implementation, a food preparation may use two or more displays. InFIG.4A, the station4100has two independent displays4131,4132. The left display4131may provide guidance for a first person to prepare the left pizza4121, and the right display4132may provide guidance for a second person to prepare the right pizza4122although not limited thereto.

Camera

The system includes one or more cameras150for capturing images of the table110and the array120. Referring toFIG.2B, a camera152is installed for monitoring food ingredients in the pans320, and another camera151is installed for monitoring the pizza220being prepared on the table110. In an implementation, a single camera may monitor both of the table110and the food pan320. In the station ofFIGS.4A to4C, a camera4151is provided for monitoring food preparation on the table4120and another camera4152is provided for monitoring food ingredients in the array4110.

Camera Location

The camera ofFIG.2Bis installed over the food pan array120and the display130to not interfere the person's sight or action. InFIG.4, the two cameras4150are installed over the displays4131,4132and the food pan array4110. In implementations, a camera system may be at a location different from the examples.

Additional Monitoring Devices

In an implementation, the station100includes a device other than a camera to monitor food ingredients or the pizza220being prepared. For example, one or more thermometers may monitor temperature of each food ingredient or the pizza. A weight measurement system can be used to measure the weight of the pizza220or a food ingredient contained in a food pan. A laser scanner or a light detection and ranging (LIDAR) device may be used for measuring a thickness of a food ingredient (e.g., pizza dough, cheese over the pizza dough) or for measuring location and distribution of an ingredient on the pizza220. In an implementation, a device other than the examples may be used.

Computing System

The computing system160is for process information relating to operation of the station100. The computing system160is connected to the display130, the light indicators140, the camera150, the database170and the ID card reader180. The computing system160may communicate with a device outside the station100. In an implementation, the computing system160can be outside a kitchen where the food preparation table110is located, and communicates with other devices of the station100via a communication network. In an implementation, the computing system160communicates with another computing system to obtain information of an order for a pizza. In an implementation, the computing system160can use computing power of another system (e.g., cloud computing). An example architecture of one or more computers systems for use with one or more implementations will be described in detail with reference toFIG.19.

Database

The database170is for storing data for providing food preparation guidance. The database170may be one or more of a local data store of the computing system160and a remote data store connected to the computing system160via a communication network. The database170may store a plurality of recipes that may be prepared at the station, profiles of worker or person, and history of food preparation works done at the station100. For each recipe, the database170may store information of necessary ingredients, and locations of the ingredients. For each worker or person, the database170may store a skill level for each pizza and history of food preparation works. The database170may store additional data other than the example, and may not store one or more of the examples. Data stored on the database170will be described in detail with reference to other drawings.

ID card Reader

The ID card reader180is for check-in and check-out of the person210at the station100. The station may include100includes one or more of an ID card reader, a keypad, and a face recognition system. The station100may include a device other than the example devices.FIG.4Ashows two ID card readers4181,4182installed on a frame of the array4110.

Providing Food Preparation Guidance

FIG.5is a flow chart for providing guidance to prepare food, here a pizza. In response to an assignment to prepare a pizza at the station100, the system may retrieve data of a worker or person, retrieve recipe data of the ordered pizza, and provide guidance according to the retrieved recipe data.
Retrieving Worker Data (S510)

In response to a check-in of the person or worker210or upon initiation of ***, the computing system160may locate the person's profile on the database170. The computing system may load data of the located profile on its local memory, or may use data already stored on its local memory without newly retrieving data from the database170. An example profile of a worker will be discussed with reference toFIG.6C. This step is optional and may be omitted.

Retrieving Recipe (S520)

In response to an order for the pizza220or upon initiation, the computing system160locates the pizza's recipe on the database170and loads data of the recipe on a local memory. This step S520may precede the step of retrieving worker data S510. The two steps S510, S520may be performed in parallel. In an implementation, the computing system160uses data stored on its local memory without newly retrieving recipe data from the database170. An example recipe (pepperoni pizza) will be discussed with reference toFIG.6A.

Providing Guidance (S530)

Based on the recipe data and the person's profile, the system may provide a food preparation guidance to the person210. For example, the system may display a text instruction on the display130, play an audio or video guide, and turn on a light indicator to notify location of a pizza ingredient. The system may provide different instructions based on the person's experience level or work history related to the current recipe. Example data for use in providing food preparation guidance will be described in detail with reference toFIG.6AtoFIG.6C.

Recipe Data

FIG.6Ashows data of an example recipe stored on the database170.FIG.6Bshow an example food preparation history.FIG.6Cshows example data of a worker (a station user). According toFIG.6A, the database stores, for each recipe, recipe name610, step number620, instruction630, ingredient640and step completion requirement650. According toFIG.6B, the database stores a log of completed orders. For each order, the database stores an order number681, a recipe name610, a Worker ID670, Time of Order Received682, Time of Order Completed683, and Preparation Speed Rating684. According toFIG.6C, the database stores profiles of workers. For each worker, the database stores a worker ID670, one or more recipes610, a preparation time rating681, and a preparation quality rating682, and an experience level680. In implementations, the database stores data in a way different from the example ofFIG.6AtoFIG.6C. The database170may store additional data different from the example, and may not store one or more of the example data.

Recipe Name (610)

The recipe name610is for uniquely identifying each recipe on the database170. When an order for ‘pepperoni pizza’ is received, a corresponding recipe600can be located using the recipe's name610. In an implementation, information other than the name of pizza may be used. For example, a predetermined code of a pizza may be used for delivering order information to the computing system160, and the computing system160locates a corresponding recipe using the predetermined code.

Sequence Number (620)

The example recipe600of ‘pepperoni pizza’ has four steps in total. Each step is numbered according to its order in the recipe, from Step 1 to Step 4. A recipe may have steps fewer or more than four. The database170may store the step order in a way different from the example ofFIG.6A.

Instruction (630)

For each step of the example recipe600, the database may store one or more instructions to help the person210during each of the recipe steps. The instructions may include one or more of a text message, an audio message and a video guide predetermined for the recipe step. For example, when the person210needs to perform Step 1 (preparing a dough), the system may locate a first message631linked to Step 1 and deliver the first message to the restaurant worker.

Text Instructions

In an implementation, the first message631includes a text instruction “Prepare a 10-inch dough”, the second message632includes a text instruction “Place sauce on ¾ of dough”, the third message633includes a text instruction “Place cheese to cover 90% of sauce”, the message534includes a text instruction “Place 12 slices of pepperoni”. These text messages may be presented on the display130to guide a restaurant worker.

Audio and Video Instructions

In an implementation, the database stores an audio or video instruction for a recipe step, and the system plays the audio/video instruction at the beginning or during the recipe step. For example, when Step 1 is completed, the system delivers a voice instruction saying “Place sauce on ¾ of dough” for Step 2. For another example, during Step 2, the system may play a video guide showing how to apply sauce repeatedly on the display130.

Selective Instructions Based on Monitoring of Food Preparation

In implementations, among instructions stored on the database170, the system may provide one or more instructions selectively based on monitoring of the pizza220. The system may select one or more instructions among a set of predetermined instructions based on one or more features identified from monitoring of the pizza being prepared. In implementations, the system may generate a new instruction that is suitable for the current status of the pizza220. For example, during Step 2 (adding sauce), the system may request to add more sauce when it is determined the amount of added sauce is not sufficient to complete Step 2.

Ingredient (640)

For each step of the recipe600, one or more ingredients are linked on the database170. For example, Step 1 for preparing a dough is linked to ‘dough’, and Step 2 for adding sauce is linked to ‘sauce’. In an implementation, no ingredient may be linked to a recipe step when the step does not involve addition or removal of an ingredient.

Completion of Recipe Step (650)

For each step of the recipe600, the database170stores one or more requirements to determine whether the step is completed. The requirements may include one or more of (1) a desirable amount or count of an ingredient to be added (or removed) during the current step, (2) a size of an ingredient on the pizza220, (3) a shape of the ingredient, (4) a desirable position of the ingredient, (5) distribution of the ingredient, (6) distance between individual pieces of the ingredient, (7) a temperature of the pizza220, (8) a predetermined time limit of the current step, and (9) a quality or status of the ingredient (e.g., freshness, frozen, melt, chopped, deformation). For example, the system may determine that Step 4 (adding pepperoni) is completed when at least 12 slices of pepperoni (each sized greater than a predetermined minimum size) are added on the pizza220. In an implementation, a requirement different from the examples may be used to determine a completed step.

Evaluating Preparation Quality of Recipe Step

In an implementation, the system may evaluate the quality pizza preparation for each of the recipe step. To evaluate the preparation quality, the system may consider one or more features discussed above for determining step completion. In an implementation, the system may evaluate a recipe step using one or more criteria different the step completion requirements. For example, the system may compute a rating for Step 4 (adding pepperoni) based distribution of pepperoni slices on the pizza220when completion of Step 4 is be determined based on the count of the pepperoni slices. In an implementation, the database170may store one or more criteria to evaluate a preparation quality of the pizza220for each recipe step.

Work History

The database170may stores records of orders prepared (or bring prepared) at the station100. As shown inFIG.6B, the database170may store, for each order, one or more of an order number681uniquely identifying the order, the name of ordered pizza610, an identification670of a person who prepared the ordered pizza, a time when the order is received682, a time when the ordered pizza is prepared683, and a speed rating of pizza preparation work684. In an implementation, the database170may store a data different from the examples ofFIG.4. In an implementation, the database170may store pizza orders prepared at a station other the station100

Worker ID (670)

The database170may stores a worker ID that is uniquely identifying a worker on the database. When a person taps his ID card to the card reader180, the computing system may obtain the person's ID (HKL) and locate data of the person on the database. In an implementation, as shown inFIG.6B, a worker ID is linked with orders681the worker prepared such that the worker's performance or experience level may be determined based on the person's order history.

Preparation Speed Rating (684)

The system may compute, for each completed order, a rating that represents how fast the ordered pizza had been prepared. The system may compute a preparation time of the ordered pizza using the order received time682and the pizza completion time683, and compares it with a predetermined desirable preparation time for the ordered pizza to determine the speed rating684. The system may measure the preparation time of the pizza from the start of the first recipe step on the table. In an implementation, the system may measure a completion time and evaluate preparation speed for each recipe step.

Worker Profile

InFIG.6C, the database170stores a profile for each worker of the station100. For each worker, the database170may store one or more of a Worker ID670, recipe names610of pizzas the worker prepared, a preparation speed rating684representing the worker's pizza preparation speed, and a preparation quality rating685representing the worker's work quality, and an experience level690of the worker. In an implementation, the database170may store data different from the examples.

Preparation Quality Rating (685)

The system may compute a preparation quality rating representing how properly the worker prepared pizzas in accordance with their predetermined recipes and quality standards. For example, for each recipe of pizzas a worker prepared, the system may evaluate preparation quality for each individual step of the recipe, and compute a percentage of steps satisfying a predetermined quality standard. The preparation quality rating685can be determined in a way different from the example.

Experience Level (690)

The database170may store an experience level for each recipe linked to the worker ID670. The experience level for a recipe may be determined based on one or more of the number of pizzas the worker prepared using the recipe, the worker's preparation time rating684, and the worker's preparation quality rating685. The experience level may be determined considering another factor different from the examples.

Different Instructions for Different Experience Levels

In an implementation, in providing guidance to prepare the pizza220, the system may consider the profile of the person210preparing the pizza220at the station100. The system may provide different instructions based on one or more of the person's experience level690and the ratings684,685about the ordered pizza (its recipe). For example, the system may provide no or limited guidance when the worker is well experienced about the ordered pizza, and may provide a more detailed guidance when the worker has a lower level of experience about the ordered pizza.

Updating Food Ingredient Location

The kitchen system indicates the location of an ingredient within the pan array while food is being prepared. To inform the location, the system needs to have the current location of the necessary ingredient, and the specific light indicator associated with the current location of the ingredient. The system performs a process to keep data current for notifying the locations of food ingredients within the pan array.

Process of Updating Ingredient Locations

FIG.7is an example process to update locations of food ingredients. The process includes capturing images of the food pan array (S710), processing captured images to determine the location of each food ingredient (S720), determining one or more indicators associated with the location of each food ingredient (S730), storing association between food ingredients and light indicators on the database170(S740).

Capturing Images of Food Pan Array (S710)

At least one camera captures images of the array120. The images of the array120may be captured continuously, periodically or intermittently. The captured images are then sent to the computing system160(or another computing device) for further processing. In implementations, the camera150may acquire a video of the array120continuously, and send at least part of the video frames to the computing system of another computing device.

Identifying Ingredients in Pans

The computing system160may process one or more images of the array120to identify food pans and food ingredients. In implementations, the computing system160with appropriate software processes one or more images to locate each food pan in the images. In implementations, the computing system160may perform image segmentation of camera image(s) using a machine-trained model, and identify one or more food pans (or food ingredients) corresponding to segment(s) in the camera images(s). In implementations, for each identified food pan, the computing system may compute one or more features (e.g., color, shape, and size, volume) of its contained material, and determine that a particular ingredient is contained in the pan when the computed feature(s) match the ingredient's feature(s) stored on the database. The system may identify food pans or food ingredients using an approach different from the examples.

Determining Location of Ingredient (S720)

The computing system160determines location of each food pan (or food ingredient) identified from processing of the images of the array120. In implementations, the computing system160may process the images of the array120to determine a reference (e.g., a corner point, a center point) for each pan and to compute a coordinate of the pan's reference point from a reference point of the frame310(e.g., a corner point, a center point). The computing system160may store the computed coordinate on the database170as the location of the pan's food ingredient. In implementations, when food pans are arranged columns and rows as inFIG.3, the system may store the location of the pepperoni pan321as Row2, Column2as shown inFIG.8.

Determining Indicator Corresponding to Ingredient (S730)

The system may determine one or more indicators that will draw attention to a particular food pan based on positional relationship between the indicator and the ingredient. Referring toFIG.3, the light indicators142,144are installed on the frame according to a predetermined layout. The location of the pepperoni pan321(Row2, Column2) is determined from processing of camera images. The system may assign the indicator141to the pan321as no other indicator is closer to the pan321and no other pan is closer to the indicator141. In implementations, the system may associate an indicator with a pan when they are within a predetermined distance from each other although not limited thereto. In implementations, the system may use a map of food pan array that defines one or more indicator assignment zones. For each zone of the food pan array, the system assigns at least one light indicator based on positional association between the zone and the indicator such that turning on the indicator would draw the person's attention to the zone. When it is determined that an ingredient (or a pan) is located at an indicator assignment zone, the system associates or links, on the database, the ingredient (or the pan) to the indicator assigned to the zone such that the indicator may be turned on to indicated location of the ingredient.

Updating Database to Store Indicator Associated with Ingredient (S740)

The system may store on the database170information of which light indicator is associated with which food ingredient. Each food ingredient may be linked to at least one light indicator on the database. InFIG.8, for example, cheese is linked to the location of the cheese pan324(Row1, Column3) which is linked to the light group147, and accordingly cheese is linked to the light group147. Based on this association between cheese and the light group147, the system may operate the light group147to indicate the location of cheese in the array120.

Updating Pan Location Changes Real Time

In implementations, the system may perform the process ofFIG.7continuously, periodically or intermittently to maintain the database170current and to reflect a pan location without delay. The system may perform the process independent of providing step-by-step instructions for the pizza220. The system may perform the process while it is providing instructions to prepare the pizza220such that the system can update the database real-time in response to a pan location change during the preparation of the pizza. The system may perform the process during a waiting time after completing a pizza such that a pan location change is reflected on the database before preparing another pizza.

Responding to Location Change Due to Food Pan Refill

Sometimes, location of a food pan may be moved in the food pan array120after refilling the food pan. For example, when the person210refills the sauce pan323and the cheese pan324after preparing a first pizza, the person210by mistake may switch locations of the two pans. In response to such pan location change, based on processing of camera images(s), the system updates the database such that the sauce pan323is linked to the light147and the cheese pan is linked to the light146. Subsequently when the person210prepare a second pizza, the system may turn on the light147when sauce is need for the second pizza while it turned on the light146when sauce was need for the first pizza.

Monitoring of Additional Feature—Ingredient Amount

Besides monitoring locations of food ingredients, the computing system160may processes one or more images from the camera150to monitor amount (for example, volume) of each food ingredient. The system may determine whether there are enough ingredients in the food pans considering one or more of a received order, an expected order, and a predetermined amount. When it is determined that a food pan does not store enough food ingredient, the system may provide an instruction to refill the food pan. In an implementation, the system may use a weight sensor, a LIDAR system, or another sensor other than the camera system for monitor amount of a food ingredient.

Step-by-Step Food Preparation Guidance

FIG.9is a flowchart of providing a step-by-step food preparation guidance based on the example recipe600. The system may provide guidance for each step sequentially from the first step (Step 1) to the fourth step (Step 4). Operation of the system for each step will be described in detail referencing to other drawings.

Providing Guidance of Individual Recipe Step

FIG.10is a flowchart of providing guidance for an individual step of a recipe according to an implementation. The process may include providing one or more instructions of the current step (S1010), indicating location of an ingredient necessary for the current step (S1020), and determining if the current step is completed based on monitoring of the pizza220being prepared (S1030). The process ofFIG.10will be explained below using the example recipe600.

Providing Instruction of Current Step (S1010)

The system may locate one or more instructions630linked to the current step on the database170, and provide the instructions to the person210working at the station100. For example, for Step 1 (preparing dough), the system may retrieve the message631linked to Step 1 from the database170, and control the display130to present the retrieved message. InFIG.12, the text instruction “Prepare a 10-inch dough” is presented on the display130for Step 1.

Activating Indicator Associated with Ingredient of Current Step (S1020)

The system may locate, on the database170, one or more light indicators linked to an ingredient necessary for the current step. To indicate the location of the necessary ingredient, the system may turn on the one or more light indicators, and turn off other indicators that are not linked to the necessary ingredient. For example, for Step 4 (adding cheese), the system refers to the database170shown inFIG.8to locate the light group146that is linked to ‘cheese’. Then, the system may turn on the segment146of the light strip to indicate location of cheese in the food pan array100.

Determining Step Completion (S1030)

For each recipe step, the system may determine whether the current step is completed to move on to the next step. The system may locate one or more completion requirements650of the current step from the database ofFIG.6A, and may determine the current step is completed when the requirements are satisfied. For example, the completion requirement for Step 4 is to add at least ‘twelve’ slices of pepperoni. The system may process one or more images of the pizza being prepared, count pepperoni placed, and determine that Step 4 is completed when the count reaches twelve. An example process for determining step completion will be described in more detail referencing toFIG.11.

Completion of Recipe

In an implementation, when it is determined that the current step is completed, the system turns off indicator lights activated for the current step, and proceeds to provide guidance for the next step of the recipe. The system may provide a notification that the current step is completed. In an implementation, when it is determined that the last step is completed, the system provides a notification that the pizza is ready for serving to a customer or ready for a further processing. An example screen ofFIG.17shows a notification that all steps at the station100are completed and the pizza220is ready to bake.

Determining Completion of Individual Recipe Step

Determining Based on Monitoring of Pizza

FIG.11shows a flowchart of determining completion of a recipe step based on monitoring of a pizza being prepared. The process may include capturing images of the pizza220being prepared (S1110), processing the images to identify one or more ingredients on the pizza220(S1120), computing a progress index of the current step (S1130), determining whether the current step is completed (S1140), and repeating the steps (from S1110to S1140) when the current step is not completed.

Capturing Images of Pizza Being Prepared (S1110)

One or more cameras may be used to monitor a dish being prepared. Referring toFIG.2B, the camera151may, periodically or intermittently, capture images of the pizza220and send the images to the computing system160or another computer for further processing. The camera151may acquire a video of the table110continuously, and send one or more frames of the video to a computing device for further processing.

Image Processing to Identify Food Ingredient (S1120)

The system may process one or more images from the camera150to identify one or more food ingredients on the pizza220being prepared. In an implementation, the computing system160detects an object in an image, determines feature(s) (e.g., color, shape, and size) of the object, and determines a food ingredient when the object's feature(s) matches the food ingredient's data stored on the database. The computing system160may use various algorithms other than the examples for identifying food ingredients. In an implementation, the computing system160uses a machine-trained model for identifying food ingredient(s) from the camera image(s). For example, the computing system may perform image segmentation of a camera image to find one or more segments each corresponding to an object in the image, to find boundaries separating the segments, and to classify pixels of the images into the segments.

Determining Visible Features of Food Ingredients

In an implementation, the system may process the camera image(s) to determine one or more features for each food ingredient appearing in the camera image(s). For each ingredient, the system may determine one or more of size, count, location and color although not limited thereto. For example, for Step 1 (preparing dough) of the example recipe, the system may compute a size, an area and a color of the dough for use in determining completion of Step 1. For Step 4 (placing 12 slices pepperoni), the system may determine one or more of the number of pepperoni slices added on the pizza220, the size of each pepperoni slice, and the location of color each pepperoni slice.

Determining Non-Visible Feature

In an implementation, the system may determine one or more non-visible features not relying on visual of food ingredients in the camera images. For example, the system may obtain one or more of the temperature of the pizza, the weight of the pizza, and time elapsed for the current step although not limited thereto.

Determining Progress Index (S1130)

In an implementation, the system may compute an index (measure) representing progress of the current step using one or more features obtained from monitoring of the pizza220being prepared. The progress index may be based one or more of the visible features, one or more of the non-visible features, and combination of thereof. Example progress indices will be discussed in detail with reference toFIG.12AtoFIG.16.

Determining Step Completion (S1140)

The system may determine the current step's completion when the current step's progress index reaches a predetermined threshold (e.g., 100%). The system may determine the current step's completion when the completion requirement650of the current step is satisfied. Once it is determined that the current step is completed, the system starts to provide guidance for the next step.

Step-by-Step Guidance for Example Recipe

Screen for Dough Preparation Step

FIG.12Ais an example screen1200for Step 1 (dough preparation) of the example recipe600.FIG.12Bis a photograph of an example pizza dough. InFIG.12A, the screen1200presents the pizza's name1210, the current step's number1220, a text instruction for the current step631, an image (or a video stream)1230of the pizza being prepared, a progress indicator1240, and time elapsed for the order1260.

Progress Based on Size of Dough

Step 1 is to prepare a ‘10-inch’ dough. The system may process one or more images of the dough1250to compute the dough's size (e.g., length, diameter, 2-dimensional area). The system may compute progress of Step 1 using the computed dough size. InFIG.12A, the current progress of 90% is computed as a ratio of the computed dough's size (9 inches) with the required size (10 inch) for completing Step 1 although not limited thereto. In an implementation, the system may consider one or more of the dough's shape, 2-dimensional area, thickness, freshness and color to determine progress of Step 1 although not limited thereto.

Completion of Dough Preparation Step

The system may determine completion of Step 1 when the dough's size satisfies Step 1's predetermined requirement. In an implementation, when a pre-baked dough is used for the pizza220, the system may determine completion of the dough preparation step when the pre-baked dough is placed on the table110. After determining completion of Step 1, the system starts to provide guidance for the next step in the recipe, Step 2.

Screen for Sauce Adding Step

FIG.13Ais an example screen1300for Step 2 (applying sauce) of the example recipe600. Referring toFIG.13A, the screen presents an image1330featuring the dough1250prepared at Step 1 and sauce1350applied over the dough. The screen may also present an instruction632for Step 2 and a progress indicator1340.FIG.13Bis a photograph of an example pizza dough with sauce added.

Progress Based on Area of Sauce

Step 2 is to apply sauce over ¾ of the dough prepared at Step 1. The system may process one or more images of the pizza being prepared to compute a 2-dimensional area of the dough1250and a 2-dimensional area of the sauce1350placed on the dough. Using the computed areas, the system may compute a ratio of the sauce area to the required area (¾ of the dough area) as the progress measure of Step 2. In an implementation, the system may compute the dough's area assuming the dough is in a circular shape and using the diameter of the dough. In an implementation, as shown inFIG.13B, the system may draw a box1371surrounding a dough1372, and may use the box's area for computing the progress measure. The system may use a processing different from the examples.

Image Segmentation to Identify Sauced Area

In implementations, the system may process the image1330using a machine-trained model to identify a first group (segment) of pixels as the sauced area1350and to identify a second group (segment) of pixels as the dough1250that is not cover with the dough. The system may compute an area of the sauced area1350using the number of pixels in the first group, compute an area of the dough using on the number of pixels in the second group, and compute a ratio between the two areas for evaluating progress of Step 2. If the first group (sauce) is of 600 pixels in the image1330and the second group (dough not covered with the sauce) is of 400 pixels, the system may determine that 60% of the dough is covered with the sauce.

Completion of Sauce Placing Step

The system may determine completion of Step 2 when the sauced area1350is larger than a predetermined percentage of the 2-dimensional area of the dough. In an implementation, the system may determine completion of Step 2 using a criterion other than the area ratio.

Example Screen for Cheese Adding Step

FIG.14Ais an example screen1400for Step 3 (adding cheese) of the example recipe600. Referring toFIG.14A, the screen presents an image1430featuring the dough1250prepared at Step 1, the sauce1350applied at Step 2, and cheese1450added over the dough. The screen also presents the instruction633for Step 3 and a progress indicator1440.FIG.14Bis a photograph of a pizza when cheese is being added.FIG.14Cis another photograph showing a cheese adding process.

Computing Progress of Cheese Adding Step

Step 3 is to place cheese to cover 90% of sauce. The system may process one or more images of the pizza being prepared to compute a 2-dimensional area of the sauce1350and a 2-dimensional area of cheese added the sauce1350. The system may compute a ratio of the area of cheese to the area of the sauce as the progress measure1440of Step 3. A different process may be used to compute the progress measure.

Virtual Grid to Compute Progress of Cheese Adding Step

In an implementation, the system may use a grid of virtual segments to determine how much cheese is placed on the sauce1350. InFIG.14A, the system overlays the grid1470over the sauced area1350to virtually partitioning the sauced area into a plurality of sauced segments1471. For each unit segment, the system determines whether it is covered with cheese or not, counts the number of cheese-covered segments, and computes a ratio of the cheese-covered segments to the entire sauced segments as the current progress1440of Step 3. In determining a cheese-covered segment, the system identifies a cheese-covered portion inside a segment based on the color of cheese and the color of sauce, and determines the segment is a cheese-covered segment when the cheese-covered portion is greater than a predetermined percentage of the segment area. In an implementation, the system identifies compute a representative color (e.g., average) of the segment, and determine the segment is a cheese-covered segment when the average color is closer to that of the cheese although not limited thereto. InFIG.14B, each of the green boxes1472represents a cheese-covered segment. In an implementation, the system may compute a progress index of Step 3 using a process different from the example.

Image Segmentation to Identify Cheese

In implementations, the system may process the image1430using a machine-trained model to classify a first group (segment) of pixels as cheese, a second group (segment) of pixels as sauce. The system may count the number of pixels for each group in the image1430(or its modified version), compute a 2-dimensional area for each group, and determine progress of Step 3 using the pixel counts and the computed areas. For example, if the first group (cheese) is of 300 pixels in the image1430and the second group (sauce on the dough) is of 700 pixels, the system may determine that 30% of the sauce is covered with the cheese.

Completion of Cheese Adding Step

In an implementation, the system may determine completion of Step 3 when cheese is placed more than a predetermined percentage of the 2-dimensional area of the pizza dough or a sauced area within the 2-dimensional area (when the computed progress reaches 100%) although not limited thereto. Subsequent to completion of Step 3, the system may provide an instruction to start Step 4.

Example Screen for Pepperoni Adding Step

FIG.15Ais an example screen for a pepperoni adding step. The screen1500presents a current image1530featuring the dough1250, the sauce1350, and cheese1450prepared at Step 3. The screen also presents an instruction634for Step 4 and a progress indicator1540.FIG.15Bis a photograph of a pepperoni pizza being prepared.

Progress Based on Counting of Pepperoni

Step 4 is to add 12 slices of pepperoni over the cheese place at Step 3. The system may process a current image of the pizza to identify pepperoni slices and to count pepperoni slices added over the cheese. InFIG.15A, the current progress of Step 4 (50%) is computed as the ratio of the current number of pepperoni slices (six) to the predetermined number (twelve) although not limited thereto. In an implementation, the system may count a pepperoni slice when it is greater than a predetermined size. The system may not count a pepperoni slice when it does not meet a predetermined requirement for pepperoni.

Determining Completion of Pepperoni Adding Step

The system may determine completion of Step 4 when the count of pepperoni slices reaches the predetermined number of twelve although not limited thereto. Subsequent to completion of Step 3, the system may provide an instruction to bake the pizza (FIG.17).

Progress Index when Food is not Fully Visible

FIG.16shows another example screen1600of Step 4 that is subsequent to the screen1500. InFIG.16, a hand1610is adding the seventh pepperoni slice1670to the pizza of the image1530(having 6 pepperoni slices), but only five pepperoni slices are visible in the image1630. If a progress index of Step 4 is computed based on the number of currently visible pepperoni slices, the progress should lower than the 50% shown inFIG.15A. It may confuse the person210if the system lowers the progress index real-time when a hand is obstructing the camera's view. To avoid such confusion, the system may not update a progress index when the pizza being prepared is not fully visible. In an implementation, the computing system160processes a camera image to determine the food being prepared is fully visible in the image, and does not consider the image for computing a progress index or evaluating a food preparation quality when the pizza is not fully visible.

Computing Progress Using Machine-Trained Model

For example, the system uses a machine-trained model to compute a progress for a recipe step and to determine completion of the recipe step. In an implementation, the system may train a model such that the model outputs a progress index of a recipe step in response to an input of an image of a pizza being prepared. For example, the system uses a machine-trained model configured to determine completion of Step 3 in response to an image featuring cheese covering a sauced dough.

Recipe Completion Message

When the last step of a current recipe is completed, the system may present a screen that the food is ready for serving or for a further processing.FIG.17is an example screen1700notifying that a pizza prepared at the system is ready to bake.

Performance Feedback

FIG.18is an example screen1800provided after completing all four steps of the example recipe. The feedback screen1800includes, for each step, (1) a first performance indices1810based on preparation time and (2) a second performance indices1820based on preparation quality. In an implementation, the system may provide an additional performance index, and may not provide one or more of the example performance indices.

Performance Rating Based on Preparation Time

In implementations, when a person performs each step of the recipe, the system collects data to evaluate the person's performance for each step. For example, the system measures a completion time for each step, compares the measured completion time with a predetermined desirable completion, and computes a performance index representing how fast the worker completed the step. In an implementation, the system updates the person's preparation time rating693using the first performance indices1810.

Performance Based on Preparation Quality

In implementations, at the end of each recipe step, the system evaluates the step using one or more criteria for determining a properly-performed step. Examples of the criteria were explained in connection with example recipe data. In an implementation, for Step 2, the system computes a performance index representing how evenly the sauce spreads on the dough. In an implementation, the system updates the person's preparation quality rating693using the second performance indices1820.

Machine-Trained Model (Artificial Intelligence)

In implementations, the computing system160uses a machine-trained model for determining location of a food ingredient, and monitoring progress of a recipe step.

Machine-Trained Model for Identifying Food Ingredients

A machine-trained model of an implementation is configured to, in response to an input of data of a photographic image, output information of one or more food ingredients featured in the photographic image. In an implementation, the system may use a machine-trained model configured to perform image segmentation of a camera image for identifying objects (pans, food ingredients) in the image.

Data Set for Training Machine-Trainable Model

A data set for training of a model includes a number of data pairs. Each pair includes input data for the training machine-trainable model and desirable output data (label) from the model in response to the input data. For example, for a machine-trainable model to identify food ingredients, the input data includes an image of a predetermined size that features one or more food ingredients, and the desirable output data includes one or more identifiers (names) of the featured food ingredients. For another example, for a machine-trainable model to evaluating progress of a recipe step, the input data includes images of food being prepared, and the desirable output data includes a percentage indicating progress of a food preparation step.

Training of Machine-Trainable Model

In an implementation, a supervised learning technique can be used to prepare the machine-trained model. Any known learning technique can be applied to the training of the model as long as the technique can configure the model to output, in response to training input images, a name (identifier) of food ingredient within a predetermined allowable error rate.

Various Structure of Machine-Trained Model

In an implementation, a convolutional neural network (CNN) is used to construct the machined trained model. In general, a convolutional neural network requires a smaller number of model parameters when compared to a fully connected neural network. In an implementation, a neural network other than CNN can be used.

Computing System

General Architecture

FIG.19depicts an example architecture of a computing system160that can be used to perform one or more of the techniques described herein or illustrated in other drawings. The general architecture of the computing system160includes an arrangement of computer hardware and software modules that may be used to implement one or more aspects of the present disclosure. The computing system160may include many more (or fewer) elements than those shown inFIG.19. It is not necessary, however, that all of these elements be shown in order to provide an enabling disclosure.

Hardware

As illustrated, the computing system160includes a processor1610, a network interface1620, a computer readable medium1630, and an input/output device interface1640, all of which may communicate with one another by way of a communication bus. The network interface1620may provide connectivity to one or more networks or computing systems. The processor1610may also communicate with memory1650and further provide output information for one or more output devices, such as a display (e.g., display1641), speaker, etc., via the input/output device interface1640. The input/output device interface1640may also accept input from one or more input devices, such as a camera1642(e.g., 3D depth camera), a keyboard, a mouse, a digital pen, a microphone, a touch screen, a gesture recognition system, a voice recognition system, an accelerometer, a gyroscope, a thermometer, an optical temperature measurement system, a sonar, a LIDAR device, a laser device, etc.

Software—Computer Program Instructions

The memory1650may store computer program instructions (grouped as modules in some implementations) that the processor1610executes in order to implement one or more aspects of the present disclosure. The memory1650may include RAM, ROM, and/or other persistent, auxiliary, or non-transitory computer-readable media. The memory1650may store an operating system1651that provides computer program instructions for use by the processor1610in the general administration and operation of the computing system160. The memory1650may further include computer program instructions and other information for implementing one or more aspects of the present disclosure. In one implementation, for example, the memory1650includes a user interface module1652that generates user interfaces (and/or instructions therefor) for display, for example, via a browser or application installed on the computing system160. In addition to and/or in combination with the user interface module1652, the memory1650may include an image processing module1653, a machine-trained model1654that may be executed by the processor1610. The operations and algorithms of the modules are described in greater detail above with reference to other drawings.

Multiple Components

Although a single processor, a single network interface, a single computer readable medium, a singer input/output device interface, a single memory, a single camera, and a single display are illustrated in the example ofFIG.19, in other implementations, the computing system160can have a multiple of one or more of these components (e.g., two or more processors and/or two or more memories).

Other Considerations

Logical blocks, modules or units described in connection with implementations disclosed herein can be implemented or performed by a computing device having at least one processor, at least one memory and at least one communication interface. The elements of a method, process, or algorithm described in connection with implementations disclosed herein can be embodied directly in hardware, in a software module executed by at least one processor, or in a combination of the two. Computer-executable instructions for implementing a method, process, or algorithm described in connection with implementations disclosed herein can be stored in a non-transitory computer readable storage medium.

Other Considerations

Although the implementations of the inventions have been disclosed in the context of certain implementations and examples, it will be understood by those skilled in the art that the present inventions extend beyond the specifically disclosed implementations to other alternative implementations and/or uses of the inventions and obvious modifications and equivalents thereof. In addition, while a number of variations of the inventions have been shown and described in detail, other modifications, which are within the scope of the inventions, will be readily apparent to those of skill in the art based upon this disclosure. It is also contemplated that various combinations or sub-combinations of the specific features and aspects of the implementations may be made and still fall within one or more of the inventions. Accordingly, it should be understood that various features and aspects of the disclosed implementations can be combined with or substituted for one another in order to form varying modes of the disclosed inventions. Thus, it is intended that the scope of the present inventions herein disclosed should not be limited by the particular disclosed implementations described above, and that various changes in form and details may be made without departing from the spirit and scope of the present disclosure as set forth in the following claims.