Patent Description:
Depth sensors exist to measure the amount of inventory that's available to serve, but these are typically complex, requiring a computer scientist or engineer to determine which sections of the pixel data to measure. Further, these sensors need to be calibrated in order to provide useful information. This application describes a solution for automatically or semiautomatically setting the regions of interest and calibrating the sensors to provide information to the user that is relevant to the operations of a restaurant.

When these sensor systems are installed and functioning, sensing the available fresh inventory is only one of the inputs to the management of production scheduling. Additionally, the system must forecast demand, determine how long the existing inventory will last, and order the production of more food when the available inventory will not last longer than the production time. Furthermore, there are constraints related to the equipment used for cooking and to the available labor in the kitchen. The system must order the production of more food earlier than it otherwise would if there are constraints on the available labor or equipment.

<CIT> is concerned with systems and processes for improved management and operation of a commercial kitchen, for example, in a quick-serve restaurant. A communication hub may interconnect a production client comprising a digital device arranged to execute a production application; a preparation/cook client comprising a digital device arranged to execute an ingredient preparation application; and one or more connected kitchen appliances. A broker (hub) provides communications among the production client, the preparation client, the connected kitchen appliance and an orchestrator application. A database may be coupled to the orchestrator for storing a set of prescribed workflows and ingredients parameters for ingredients to make menu items. In an embodiment, an orchestrator client is configured to select and execute one or more of the workflows responsive to inputs received via the broker to manage preparation and holding of the ingredients to maintain menu item quality and food safety compliant with the corresponding ingredient parameters.

<CIT> describes a cooking assistance appliance, which includes a body, at least one camera, at least one sensor, and an imaging device for capturing an image of a cooking appliance. The cooking appliance can provide for monitoring use of the cooking appliance, as well as assisting the user in operating the cooking appliance. Additionally, the cooking assistance appliance can be integrated into a hood or a microwave oven and vent combination above a stovetop.

<CIT> relates to food processing and more particularly to systems, apparatus, and methods for managing food processing in food preparation establishments such as restaurants, including quick service restaurants.

An object of the present invention is therefore to provide a system or food production system or food processing system or scheduling system or tracking and scheduling system that enables efficient planning and control of food preparation or food supply.

This object is achieved by a system for processing food with the features of claim <NUM>, by a method with the features of claim <NUM>, and by a computer program with the features of claim <NUM>. Advantageous embodiments with useful further developments are given in the dependent claims. Embodiments not covered by the claimed subject-matter do not form part of the invention.

A food processing system comprises: a sensor unit for determining holding data or food holding data of at least one pan, container, or food holding container placed in a holding area or food holding area; a processing unit for determining a scheduling state or food scheduling state based on current holding data or food holding data and/or a holding data history or food holding data history; and a control unit to control an actuator based on the determined scheduling state or food scheduling state.

The system is able to control an actuator, such as a robotic process, or to inform personnel to prepare food in good time on the basis of data indicating current or previous consumption of food. This makes it possible to keep cooked food available without delay times.

The sensor unit comprises at least one of an RGB sensor, or other optical sensor, and at least one of a depth sensor, a thermal sensor, a 3D camera, a time of flight sensor, or a stereo camera. Said sensors are physically installed in a fixed position and orientation relative to each other. The combination of optical data and depth data allows tracking the depletion rate of food within the pan, container, or food holding container in detail.

In this context, the holding data or food holding data can comprise information about at least one of a fill level of the at least one pan, container, or food holding container, a holding time or food holding time of the at least one pan, container, or food holding container, a food ingredient associated with the food in the pan, container, or food holding container, information about the availability of the food ingredient, and a food ingredient preparation time. The scheduling state or food scheduling state can comprise information about the types of food that should be cooked or prepared, the quantity of food that should be cooked or prepared, the destination where the food should be brought, the priority level relative to other scheduled foods, and/or the timing of when the food should be finished cooking or preparing. A cook command is a message communicating information comprised in the scheduling state or food scheduling state, either to initiate a robotic automation process or to instruct a human operator to begin a cook or preparation process.

The system can determine the fill level based on 3D pixel data of the holding area or food holding area.

Said 3D pixel data can be calculated by the system based on correlating 2D pixel sensor data and depth sensor data, which are determined by the sensor unit.

The system can then determine regions of interest within the sensor unit's field of view based on the 3D pixel data. It is advantageous to associate measured distances or depths with fill levels of at least two regions of interest different from one another.

In addition, the system can determine a heat state fill level based on enhanced 3D pixel data of the holding area or food holding area by correlating 2D temperature sensor data with 2D pixel sensor data and depth sensor data, which are determined by the sensor unit. With said advanced information also the heat state of the food in the pan, container, or food holding container is taken into consideration.

The scheduling state or food scheduling state can be based on the current fill level and/or current heat state fill level.

Based on the scheduling state or food scheduling state, the control unit can identify a replenish event when a specific fill level of the pan, container, or food holding container is reached. It can start a timer by the replenish event for a predefined ingredient specific holding time. It can initiate cook commands to a grill, stove, fryer, toaster, ice machine, oven, coffee maker or beverage dispenser crew person or initiate an automated robotic process once a predefined time is reached. Moreover, the control unit can initiate replenish commands to exchange the pan, container, or food holding container once a predefined time is reached.

The scheduling state or food scheduling state is based on a holding data history or food holding data history, wherein, based on the scheduling state or food scheduling state, the control unit is adapted to further forecast from previous sales volume or current customer traffic the necessary food item quantity or weight to be cooked in a certain time frame. It is adapted to augment the forecasting by adding local events, weather, calendar holidays into the calculation of current and future demand. It is adapted to to identify reorder points using the information of anticipated consumption, to initiate cook commands once fill levels drop below a calculated threshold. Furthermore, the control unit is adapted to calculate different reorder points for each time of day and for each ingredient individually.

Thus, the whole food preparation and planning process can be automated saving time and costs.

The system further comprises a display unit that is adapted to display the fill levels of the pan, container, or food holding container, available inventory, specific cook commands, action items or requests of a crew person, and/or a destination area for the pan, container, or food holding container. Furthermore, the system is adapted to prioritize cook commands based on demand and available inventory.

The system can apply vision AI to monitor a grill surface or other cooking device, and identifies what food ingredients are in the process of cooking.

The above object is also solved by a computer implemented method for processing food, the method comprising the following steps:.

wherein the scheduling state is further based on a holding data history, wherein, based on the scheduling state, the method further comprises the steps of.

Furthermore, additional steps can be performed, comprising:.

The above object is also solved by a computer program comprising instructions that, when executed by a system, cause the system to execute the above described method.

<FIG> shows a simplified schematic view of a food processing system <NUM> according to an embodiment of the present invention. In particular, the present invention is related to a system <NUM> and method for aligning sensors for food production scheduling. The system <NUM> comprises a sensor unit <NUM>, a processing unit <NUM>, and a control unit <NUM>.

The sensor unit <NUM> determines holding data or food holding data of at least one pan <NUM> or food holding container <NUM> placed in a holding area or food holding area FHA. The at least one pan <NUM> or container <NUM> is representative for all possible food containers, receptacles, pan mounts and/or food storage or preparation utensils.

The processing unit <NUM> determines a scheduling state or food scheduling state based on current holding data or food holding data and/or a holding data history or a food holding data history.

Based on the determined scheduling state or food scheduling state, the control unit <NUM> controls an actuator <NUM>. The actuator <NUM> can be a crew personal such as a cook that receives cook commands from the control unit <NUM> to prepare and cook a specific food ingredient. In addition, the control unit <NUM> can also initiate an automated robotic process to prepare and cook food.

The processing unit <NUM> and the control unit <NUM> each may comprise a computer, a virtual machine or container running on a computer, or a software suite running on a computer. The processing unit <NUM> and the control unit <NUM> each may contain memory for holding data to process, processing power to make computations and execute program instructions stored in the memory.

The sensor unit <NUM>, the processing unit <NUM>, the control unit <NUM>, and the user interface or display unit may be connected via a network for moving data between them. This network may include connections such as physical ethernet, wi-fi or other LAN connections, WAN, VPN, cellular, bluetooth, or other connections for transferring data between system components and to users both onsite and remote to the system.

In the following the sensor unit <NUM> and the processing unit <NUM> are described in detail.

Identifying different pans <NUM> or pan mounts and dimensions in an image and using image coordinates to identify a normalized image or depth map that corrects different observation points in at least two different measurement setups.

Correlating an RGB sensor's <NUM> pixel coordinates with a depth sensor's <NUM> pixel coordinates to determine the pixel coordinates of an item in the depth sensor's field of view. Using computer vision or other types of processing to locate items on the RGB stream's pixel coordinates in real time, and to draw depth measurements from a pixel area defined by the processing of the RGB stream.

In general, the sensor unit <NUM> comprises an RGB sensor <NUM> or other optical sensor <NUM> and an additional sensor <NUM>, which may be any one or a combination of digital sensors including depth sensors, thermal sensors, 3D cameras, time of flight sensors, or stereo cameras, which may depend on the desired sensing result or results. The RGB sensor <NUM> or other optical sensor <NUM> may provide a stream of data or images in the form of pixel data, which can be used to find objects or target regions of interest in the sensor's field of view or pixel coordinates. The RGB <NUM> or other optical sensor <NUM> may be physically installed in a fixed position and orientation relative to the additional sensor <NUM>.

In general, the processing unit <NUM> may receive as an input the stream of data or images from the RGB <NUM> or other optical sensor <NUM>, and may perform processing by method of computer vision, algorithmic image processing, corner detection, blob detection, or edge detection or other pixel data processing method, and may send as an output a stream of images annotated with the pixel coordinates and/or pixel size of various objects or regions of interest in the sensor's field of view.

The additional sensor <NUM> may receive as an input the annotated stream of images from the processing unit <NUM>, to determine where in its field of view lie certain specified objects or regions of interest. The additional sensor <NUM> may use the annotations to determine a direction, a pixel, or a group of pixels for measurement. The additional sensor <NUM> may measure the depth, temperature, or other parameters of the direction, pixel, or group of pixels specified by the processing unit.

The preferred additional sensor <NUM> is a sensor <NUM> that can measure a distance for at least one point, including TOF, LIDAR, 3D or stereo camera setup, including illumination unit including visible light wavelength as well as infrared wavelength, including advantageously a 3D camera setup that also includes a RGB image of the same area it monitors with depth sensing.

It is advantageous to calibrate multiple observed pans <NUM> with multiple regions of interest in one image or point matrix or vector, and to associate the measured distance with fill levels of at least two different regions of interest different from one another. Individual configuration files enable the system <NUM> to set different parameters for fill level at different regions of interest.

To calibrate the sensor <NUM>, a technician places pans <NUM> or receptacles with different volumetric amounts of product, such as <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, and empty, into different pan positions. The technician confirms the fill level of each pan position on a user interface. A configuration file is created for each pan position, including the depth reading at each pan position. When the system <NUM> is activated for readings, the system <NUM> reads the depth reading at each pan position, and checks it against the calibration values in the configuration file corresponding to the selected pan position. Once several systems <NUM> in the same restaurant chain have been calibrated, the system <NUM> can be calibrated only for empty pans <NUM>, skipping the full and partially full calibration. The system <NUM> can then learn over time the typical fill level seen in operations.

Volumetric assessment using depth sensing enables the system <NUM> to measure the available inventory by measuring the fill levels of a previously calibrated pan <NUM>. The process includes methods to lower impact of occlusions, such as taking the longest distance from sensor <NUM> to food item or using object recognition to identify utensils such as spoons on a corresponding image to carve out a region of interest in a point matrix representing a depth measurement, and/or calculating the average depth value of pixels identified as not being outliers. Volumetric sensing is converted to a % fill level for each pan <NUM>. Correlating the volumetric assessment with food item ingredient data and preparation specific food density, such as for instance a sliced meat to calculate the available food item weight according to reference weights and the current volumetric fill level.

The measured fill levels of the pans <NUM> or containers or food holding containers <NUM>, which are indicated by the dotted circles in <FIG>, are comprised in the holding data or food holding data. Said data further includes information about a food holding time of the pan <NUM> or container <NUM>, a food ingredient associated with the food in the pan <NUM> or container <NUM>, information about the availability of the food ingredient, and a food ingredient preparation time.

In this context, <FIG> shows a simplified schematic view of the holding area or food holding area FHA comprising pans <NUM> or containers <NUM> according to an embodiment of the present invention. The regions of interest for three pans <NUM> or containers <NUM> in this example are indicated by a dashed rectangle. As described above, the occlusions created by the spoons are filtered out of the processed data.

It's beneficial for the system <NUM> to determine the fill level based on 3D pixel data of the holding area or food holding area FHA. Said 3D pixel data is calculated based on correlating 2D pixel sensor data and depth sensor data, which are determined by the sensor unit <NUM>.

The system <NUM> then determines regions of interest within the sensor unit's <NUM> field of view based on the 3D pixel data. In the example shown in <FIG>, the field of view is indicated by the dashed lines. It is advantageous to associate measured distances or depths of the depth sensor data with fill levels of at least two regions of interest different from one another.

In addition, a heat state fill level is determined based on enhanced 3D pixel data of the holding area or food holding area FHA by correlating 2D temperature sensor data with 2D pixel sensor data and depth sensor data, which are determined by the sensor unit <NUM>.

The scheduling state or food scheduling state can be based on the current fill level and/or current heat state fill level. Depending on said state specific food cooking commands can be issued by the control unit <NUM>.

Based on the depth reading, pixel coordinates, and pixel size of a pan, a scoop, or a specific serving, the system <NUM> can calculate the weight or volume of food that was removed from the pan <NUM> or warm holding container <NUM>. By calculating the size of a serving of food, the system <NUM> can present information to management about the rate of depletion for each ingredient. The system <NUM> can identify automatically whether a single or double serving of an ingredient was provided.

Depth readings may be adjusted based on the pixel coordinates of the reading. The distance from the sensor <NUM> to the measured position may be multiplied by the cosine of the effective angle Θ to determine the closest distance from the measured point to the plane of the sensor <NUM>. This calculation may be done on a per-pixel basis or for discrete identified regions of interest to reduce the required processing power or CPU load for calculation.

<FIG> shows an example for adjusting said depth readings according to an embodiment of the present invention. The depth reading for each pan <NUM> or food holding container <NUM> can thus be corrected.

It is advantageous to use this method to have a unified observation over multiple locations for a grill or grill warm holding area FHA. The system <NUM> may track items with object detection including differentiation by caliber (burger patty thickness) and associating timers for each observed grill item. The system <NUM> may check if timers exceed predefined holding times, such as <NUM> minutes for a burger patty. The system <NUM> may initiate an event on a screen or a notification to a crew person once a predefined time is exceeded. The system <NUM> may calculate a target inventory of food items and initiate an event on a screen or a notification to a crew person once a predefined count or inventory is exceeded or fall below a predefined threshold. The system <NUM> may dynamically add or reduce the threshold when a customer traffic event occurs and is sensed. A customer traffic event may be a daypart-dependent predefined number to be added to the target inventory in the event a car is detected in drive thru or customer detected walking into the store. For example, a car pulling into a drive thru at 1pm may represent <NUM> burger patties, <NUM> ounces of fries, and <NUM> chicken nuggets, which may be added to the demand forecast and threshold.

The system <NUM> may use action detection to identify start and end time for a certain cooking or preparation action in a kitchen from a video stream. It is advantageous to use measured time or an average over multiple observations of the same action to calculate the current time to complete certain tasks in a kitchen such as breading chicken, distributing dough over a plate, assembling a burger or loading or unloading of a kitchen device such as a fryer or putting items into a bag or tray. It may be advantageous to use the calculated action times to forecast current or future throughput or time to prepare a certain food quantity. Associating the measured timings with expected or measured customer traffic to suggest action items to kitchen crew or adapt staffing plans or order ingredients.

For fast food restaurants, retail bakeries, fast casual restaurants, other foodservice, or other fresh food retail locations, effective production planning results in food being available and fresh at all hours of the day. To achieve this, the system <NUM> must use a sales demand prediction to determine the "reach" of each ingredient, defined as the duration that the current inventory will last. Each ingredient has a measurable call time, defined as the duration of time that it takes to prepare and bring fresh ingredients to their destination, once the system has requested that staff prepare them. When the reach of a given ingredient is less than the call time, this is an indication that the production process is behind schedule.

Forecasting from previous sales volume or current customer traffic such as walk-in, inbound digital orders or cars before order point in drive thru, the necessary food item quantity or weight to be cooked in a certain time frame, for instance the next <NUM> minutes. Forecasting accuracy can be augmented by adding local events, weather, calendar holidays, or other modifications into the calculation of current and future demand. Identifying reorder points using the information of anticipated consumption, to initiate cook commands once fill levels drop below a calculated threshold. In particular of advantage to calculate different reorder points for each time of day and for each ingredient individually. Initiating cook commands to grill, stove, fryer, toaster, ice machine, oven, coffee or beverage maker crew person or automated robotic process once fill levels are below the calculated threshold. Initiating replenish commands to exchange a pan, basket, tray, or other warm holding receptacle, once the receptacle reaches a certain pan fill level.

The scheduling state or food scheduling state is based on a holding data history or a food holding data history. Taking account of said state, and in view of the above, the control unit <NUM> can forecast from previous sales volume or current customer traffic the necessary food item quantity or weight to be cooked in a certain time frame. Moreover, it can augment the forecasting by adding local events, weather, calendar holidays, or other modifications into the calculation of current and future demand. In addition, reorder points can be identified using the information of anticipated consumption, to initiate cook commands once fill levels drop below a calculated threshold. The control unit <NUM> can also calculate different reorder points for each time of day and for each ingredient individually.

Identifying replenish events by surpassing a fill level for a pan fill level measurement over a certain time frame. Starting a timer by the replenish event for a predefined ingredient specific holding time. Initiating cook commands to grill, stove, fryer, toaster, ice machine, oven, coffee maker or beverage dispenser crew person or automated robotic process once a predefined time is reached. Initiating replenish commands to exchange pan once a predefined time is reached.

Thus, based on the scheduling state or food scheduling state, the control unit <NUM> can identify a replenish event when a specific fill level of the pan <NUM> or food holding container <NUM> is reached. It can start a timer by the replenish event for a predefined ingredient specific holding time. In addition, it can initiate cook commands to a grill, stove, fryer, toaster, ice machine, oven, coffee maker or beverage dispenser crew person or initiate an automated robotic process once a predefined time is reached. The control unit <NUM> can also initiate replenish commands to exchange the pan <NUM> or food holding container <NUM> once a predefined time is reached.

The system <NUM> may adjust its requests or cook commands by adding additional time allowances for equipment constraints. For example, a grill surface or fryer vat may be used for multiple ingredients A and B. If ingredient A has a reach of <NUM> minutes and a call time of <NUM> minutes, and ingredient B has a reach of <NUM> minutes and a call time of <NUM> minutes, the system <NUM> must order ingredient B now, since the restaurant will need both ingredients within the total call time of the two ingredients. Similarly, each cook command requires a certain measurable labor time to execute the cooking, preparation, and serving of the food. By tracking the available labor and the required labor for each cook command, the system determines when labor constraints would cause delays and can send cook commands earlier depending on the severity in minutes of the anticipated labor shortage.

Operational rules and best practices can be input into the system <NUM> as assumptions for the calculations. For example, sometimes a new batch of food will replace an old batch and the old batch can be added on top of the new batch; whereas sometimes a new batch will replace an old batch and the old batch will be thrown away. Certain ingredients can be cooked or baked together in the same oven chamber or other cooking device, while certain combinations of ingredients cannot be cooked together for food safety or other reasons. The system <NUM> plans for this by forecasting sales into the future, planning its next several steps, and combining the requirements for the next several steps.

Certain equipment can cook a linear amount of food, such as a grill surface that holds a specified amount of chicken. However, certain equipment such as fryers can cook larger or smaller batches using the same size oil vat. For example, the system <NUM> can request larger batches rather than sending multiple smaller batches, since this requires less labor and less space on the fryer or other equipment. Conversely, if the larger batch will cause overproduction and stale food, the system can request more smaller batches. Over time through supervised learning, unsupervised learning, or reinforcement learning, the system <NUM> makes an ever-improving value-based decision as to which variables are most important to the operation of the kitchen: whether labor or equipment space is the limiting factor at a given time, or whether violating hold times outweigh the equipment or labor concerns.

To include equipment constraints and calculate the effective call times, the system <NUM> adds together the call times of each ingredient that shares a piece of equipment. The system <NUM> compares this effective call time against an effective reach, calculated by adding together the reach of the ingredients that need to be cooked. The system <NUM> decides on a planned order of ingredient cooking, so that the effective call time is less than the effective reach at each intermediate step.

The call times for each ingredient are not perfectly static. Average call times may be estimated, but the true call time depends on the number of outstanding tasks for crew and the amount of staffing available to carry out these tasks. Typically all production planning will include additional time for human or robot execution of the cooking and related processes, as well as a factor of safety in case unforeseen events slow down the production process.

Fresh food has a finite holding time. One result of this fact is that the system <NUM> must request new production not only when the available inventory is running low, but also periodically to ensure the available food is within its acceptable hold times. The system <NUM> plans for these batches to be cooked by setting timers from the moment when food is sensed to have been replenished. The system <NUM> keeps a database of the parameters of each ingredient, including maximum hold times. By subtracting the actual time the food has been on the line from the maximum hold time, the system <NUM> can determine the time remaining before an ingredient must be replenished. Further subtracting from this the call time for that ingredient, the system <NUM> can calculate the ideal moment when new food production must be requested from the crew.

The system <NUM> logs its own requests, the logic for each cook command or request, and when the command was addressed in the kitchen. Measuring the time a cook command was displayed to a crew member and the time the crew member acknowledges or bumps it off the screen or can be seen by action detection that the cooking command is being started to determine a response time. Calculating an average response time over multiple observations. Calculating reorder points over multiple observations.

Using a database to count pans observed entering or exiting certain regions of interests or shelfs within a store.

The system's <NUM> calculated decisions are executed and lead to a result that either is beneficial or detrimental to the restaurant's operations. Beneficial outcomes include having products available when they are needed, discarding minimal food waste, and serving food with higher freshness than is typically observed. Detrimental outcomes include ingredients stocking out, or having a large amount of food reach its maximum hold time. The system <NUM> captures each of these signals and improves over time, using an agent such as a reinforcement learning agent. The actions and decisions leading to beneficial outcomes are rewarded, whereas the actions and decisions leading to detrimental outcomes are punished, so they happen less often in similar situations in the future.

In addition to the above, a computer implemented method for processing food according to an embodiment of the present invention comprises the following steps:.

The stereo camera sensing is in particular useful if an IR projector or an RGB LED module is added, black framing or rubber is used for isolation, and if mounted on a ceiling mount USB or Ethernet plugs or a multi axis moving plate are added to the mounting. It is further advantageous to add optical filters such as IR filter, IR passthrough, longpass filters, shortpass filters, bandpass filters, polarizing filters, or visible wavelength passthrough coatings. These can remove noise in depth readings that result from direct light reflections or other sources of data noise.

It is further advantageous if pre-processing of any image data includes the calculation of HDR images or color normalization.

Furthermore, the following steps can be performed:.

A computer program comprising instructions that, when executed by the system <NUM> as described above, causes the system <NUM> to execute the method steps indicated above.

Many ingredients take time to prepare for service, but it is not practical to do preparation during the peak hours of meal times. To solve this problem, many restaurants prepare food ahead of time and use hot or cold storage units where they can keep prepared pans, trays, or other containers of prepared food.

The system <NUM> needs to manage these storage units in order to instruct crew members when to prepare more food. To track this inventory, the system senses the amount of food that is prepared cold or hot in a storage unit, based on an RFID, QR code, OCR, barcode or other visual or code tag.

System <NUM> calculates the "reach" in time of the available inventory in secondary storage, and signals kitchen crew or management to prepare, cook, or order more of the item based on the predicted sales volume for that ingredient for the upcoming minutes or hours, or for the rest of the day or week.

Based on the sales demand forecasts, the system <NUM> can inform crew at the beginning of each day, as well as during off-peak service hours, how much food to prepare and keep in warm or cold storage. These inventory levels are monitored throughout the day, and the need for preparation is recalculated based on dynamic sales forecasts and dynamic sensing of the stored inventory.

The amount of prepared food in secondary storage is often limited by labor. The system <NUM> can manage the labor required to prepare the forecasted amount of each ingredient, by comparing the average and expected time for each task with the amount of labor available and allocating labor hours to the most pressing tasks.

The system <NUM> further comprises a display unit that is adapted to display the fill levels of the pan <NUM> or food holding container <NUM>.

User interface displays fill levels of available inventory and/or stored secondary inventory. User interface shows specific cook commands, action items, or requests of the crew.

Each food producer i.e. grill operator, produces food for multiple service channels that each need to be stocked with available inventory. UI specifies for each cook command, not only the type of product and the batch size but also where the cooked product should be sent. For example there may be hot holding at a customer-facing sales channel and at a drive through sales channel. The UI for a production crew member saves time by displaying the destination sales channel(s) for each batch.

The UI may display the cook command before it displays the destination service area. If the destination service area is designated once the food is already cooking, this gives the AI-driven system more time to see how events and inventory levels develop, and to make an appropriate decision.

Batching: UI specifies how much of the product to cook. For each batch, the UI displays a timer on the UI for when to flip or remove items from a cook process such as a grill surface.

The system <NUM> further applies vision AI.

Vision AI monitors the grill surface or other cooking device, and identifies what items are in the process of cooking. The system <NUM> automatically changes the state of these items on the UI when cooking, and removes them when cooking is complete. Vision AI also may monitor the crew members and their actions using computer vision, pattern recognition, or action detection algorithms. When the crew member completes the task to be completed, the recognized action is automatically removed from the UI's "to-do" list.

Using vision AI, the system tracks each burrito, bowl, or other menu item as it's being prepared. In other words, using vision AI, the system tracks each menu item as it's being prepared. As ingredients are added to the item, the system identifies each ingredient and "rings up" the customer for an item containing the ingredients that were served. This automatically detected checkout can be confirmed or modified by a staff member, or can be automatically accepted.

In many restaurants, the ingredients that are used affect the price that the customer should pay for the item. In these cases, the system <NUM> uses vision detection methods to identify and track both the ingredients, and to which menu item they were added. Each menu item is visually identified and the system tracks a list of ingredients that were added to it, so the software can assign a price to each item based on its ingredients.

For robustness of function, it is typically advantageous to have the detection, Ul's, and all business or operational calculations occur in devices on site at the edge. Computer vision, machine learning, or other AI models can be updated remotely via the cloud, but deployed directly on devices at the restaurant level. Other software updates can also be engineered remotely and deployed to numerous restaurants during off-hours such as late at night.

Claim 1:
Food processing system (<NUM>), comprising:
- a sensor unit (<NUM>) for determining holding data of at least one pan (<NUM>) or holding container (<NUM>) placed in a food holding area (FHA);
- a processing unit (<NUM>) for determining a scheduling state based on current holding data and/or a holding data history; and
- a control unit (<NUM>) to control an actuator (<NUM>) based on the determined scheduling state,
wherein the scheduling state is further based on a holding data history, wherein, based on the scheduling state, the control unit (<NUM>) is further adapted to:
- forecast from previous sales volume or current customer traffic the necessary food item quantity or weight to be cooked in a certain time frame;
- augment the forecasting by adding local events, weather, or calendar holidays into the calculation of current and future demand;
- identify reorder points using the information of anticipated consumption, to initiate cook commands once fill levels drop below a calculated threshold; characterized in that
- the sensor unit (<NUM>) comprises at least one of an RGB sensor (<NUM>), or other optical sensor (<NUM>), and at least one of a depth sensor (<NUM>), a thermal sensor (<NUM>), a 3D camera (<NUM>), a time of flight sensor (<NUM>), or a stereo camera (<NUM>); and
- the control unit (<NUM>) is further adapted to calculate different reorder points for each time of day and for each ingredient individually.