Patent Description:
In at least one aspect, a food storage system is disclosed. The system can comprise an appliance having a cabinet defining an interior, the interior comprising at least one of a shelf, compartment, or drawer configured to store at least one food item, a vision system comprising at least one imager configured to record image data related to the at least one food item stored within the interior, and a controller in communication with the vision system. The controller can be operable to determine an identity of the least one food item, determine a storage configuration of the interior, generate an augmented reality representation of the interior based on the determined identity of the at least one food item and the storage configuration of the interior, and display the augmented reality representation. The augmented reality representation can comprise a digital rendering of the storage configuration and a digital food item representation corresponding to the at least one food item arranged within the digital rendering.

In at least another aspect, a food storage system is disclosed. The system can comprise an appliance having a cabinet defining an interior, the interior comprising at least one of a shelf, compartment, or drawer configured to store at least one food item, a vision system comprising at least one imager configured to record image data related to the at least one food item stored within the interior, a lighting system comprising at least one light source configured to project a pattern of light, and a controller in communication with the vision system. The controller can be operable to determine an identity and a location of the at least one food item, analyze a query from a user regarding the at least one food item, and control the lighting system to project a pattern of light onto the at least one food item in response to the query. The pattern of light can provide feedback to the user indicating at least one of a location or a food item property of the at least one food item.

In at least another aspect, a food storage system is disclosed. The system can comprise an appliance having a cabinet defining an interior, the interior comprising at least one of a shelf, compartment, or drawer configured to store at least one food item, a vision system comprising at least one imager configured to record image data related to the at least one food item stored within the interior, and a controller in communication with the vision system. The controller can be operable to detect at least one food item property for the at least one food item, generate an augmented reality representation of the interior based on a user selection of a food item property, and display the augmented reality representation. The augmented reality representation can comprise at least one digital food item overlay corresponding to the food item property.

<CIT> discloses an augmented reality application which can display inventory information on a user device pertaining to items on a shelf in a refrigerator.

<CIT> discloses a refrigeration appliance provided with a system for displaying information on the cabinet's internal walls and shelves.

Referring to <FIG>, a storage system <NUM> for an appliance <NUM>, such as a refrigerator, is shown. The appliance <NUM> includes a cabinet <NUM> defining an interior <NUM>. The cabinet may include a display screen <NUM>, which can be a LCD or any suitable display. The display screen <NUM> may be in the form of a user interface. The interior <NUM> may include a plurality of holders defining a storage configuration to store at least one item <NUM>, which may include a food item <NUM>. The holders may in include at least one of a shelf <NUM>, compartment <NUM>, or drawer <NUM> and any combination thereof. While the interior <NUM> is shown to include a combination of shelves <NUM>, compartments <NUM>, and drawers <NUM>, it is not limited to these storage configurations and may include any suitable structure for storing a food item <NUM>. Furthermore, the interior <NUM> includes a lighting system including at least one light source <NUM>. The at least one light source <NUM> is configured to provide selective lighting, such as a pattern of light <NUM>, and may be in the form of any suitable light source <NUM>, such as a micro projector, a laser, and the like.

The storage system <NUM> may be configured to recognize and track a status of the item <NUM> stored within the appliance <NUM>. The specific status of the item <NUM> tracked or updated by the system <NUM> may vary depending on the nature of the appliance <NUM>. Accordingly, the disclosure may provide for a storage system <NUM> that may be utilized to recognize the status of an item, inventory of an item, and/or various processing states to gather and track various information as discussed herein.

For example, if the appliance <NUM> corresponds to a refrigerator, as illustrated in <FIG>, the system <NUM> may be configured to track an inventory of the food item <NUM> as it is added or removed from the interior <NUM>. In order to detect and identify the food item <NUM>, the storage system <NUM> comprises a vision system including at least one imager <NUM>. In some embodiments, the at least one imager <NUM> may correspond to a plurality of imagers <NUM>. The at least one imager <NUM> may be configured to capture image data in the interior <NUM> or in an access region external to the appliance <NUM>. The storage system <NUM> may be operable to detect the food item <NUM> in connection with at least one part of a person's body (e.g. a hand) in connection with the food item <NUM> entering and/or exiting the interior <NUM> via the access region. In this way, the system <NUM> may be operable to track the status of the food item <NUM> by detecting the part of the person's body as it enters and exits the access region. A system operable to track the status of a food item by detecting part of a person's body as it enters and exits the access region may include the system disclosed in <CIT>, entitled "Interaction Recognition and Analysis System," which is herein incorporated by reference in full. <CIT> discloses an object tracking system configured to track an inventory of the object as it is added or removed from an operating volume of a refrigerator.

The vision system may further include one or more infrared (IR) cameras, or sensors. An infrared camera may be incorporated into the system <NUM> in order to detect a temperature of a food item <NUM> or the temperature of various locations within the interior <NUM> of the appliance <NUM>.

The item <NUM> may generally correspond to any form of object that may be stored, processed, disposed of, or otherwise utilized in connection with an appliance <NUM>. As previously discussed, the item <NUM> may correspond to a food item <NUM>, or product, that may be stored in the interior <NUM> of the refrigerator <NUM>. The system <NUM> may process image data captured by the at least one imager <NUM> in order to identify a product type and proportion or quantity by utilizing various imaging processing techniques. With the product type and quantity identified for the food item <NUM>, the system <NUM> may update an inventory status of the product type and quantity of the food item <NUM> in a memory or inventory database. Though discussed in reference to an inventory status, the system <NUM> may be configured to detect various forms of information in reference to the food item <NUM>, which may include, but are not limited to, a depletion or usage, a location, a quality status (e.g. the presence of mold), a color or consistency, and/or various additional information that may be derived from the image data.

In some embodiments, the system <NUM> may be operable to track various forms of information regarding the status and characteristics of the food item <NUM>. As discussed herein, such information may be inferred by the system <NUM> based on a process completed by the appliance <NUM> and/or a duration of time between a first detection of the food item <NUM> and a second detection of the food item <NUM> (e.g. a time between removal and placement of the object in the operating volume, or interior <NUM>). Such information may include clock and calendar data for inventory and quality tracking of the food item <NUM>. A status or characteristic of the food item <NUM> may also be inferred by monitoring of depletion, or fill level, of the food item <NUM>. Additionally, the status of the food item <NUM> may be indicated based on actions taken on the object including, but not limited to, wrapping the food item <NUM>, opening a package containing the object, covering a surface or container of the object, etc..

Based on the image data, the system <NUM> may identify the food item <NUM>. Details regarding image processing techniques that may be utilized to identify various food items <NUM> and corresponding states of such food items <NUM> are further discussed in reference to <FIG> and <FIG>. The identification of the food item <NUM> may include an indication of a type or category to which the food item <NUM> belongs and a proportion or quantity of the food item <NUM>. The system <NUM> may further identify additional information about the food item <NUM>, including, but not limited to, a color, a texture, a storage data, a location, and various additional information. In some embodiments, the system <NUM> may classify the food item <NUM> into classes and subclasses. For example, in the case where the food item <NUM> is a food product, the system <NUM> can determine a class of the food item <NUM> as a beverage, a protein, a fruit, a vegetable, a packaged industrial food, a dressing, and the like. A subclass of a beverage may include a can, a bottle, a box, and the like, while a subclass of a protein may include a type, such as chicken, beef, fish, a state, such as frozen or fresh, and/or a size.

The system <NUM> may be operable to detect and update the status of the food item <NUM> based on a variety of properties and/or characteristics that may be identified in the image data received from the imager <NUM>. For example, as previously discussed, an identification of the food item <NUM> within the interior <NUM>, or in connection with the hand being inserted or removed from the operating volume <NUM> of the refrigerator <NUM> may be utilized by the system <NUM> to update an inventory of the food item <NUM>. The system <NUM> may track a variety of properties and/or characteristics corresponding to the food item <NUM> based on the image data captured by the imager <NUM>.

For example, the system <NUM> may detect an identity, color, surface texture, quantity, fill level, proportion, or any form of information that may be visually and/or thermally detected by the system <NUM> based on the image data received from the imager <NUM>. The system <NUM> may draw various inferences regarding the status of the food item <NUM> based on the image data. In some scenarios, the system <NUM> may also infer information about the food item <NUM> based on a duration of time that the object is detected as being removed from the interior <NUM>.

Accordingly, a controller of the system <NUM> may be in communication with an appliance control system to determine a food item property for the food item <NUM> based on one or more properties and/or characteristics identified from the image data. The controller of the system <NUM> may be configured to identify the food item <NUM> by utilizing an object identification module and determine the appropriate food item property. Furthermore, based on identification of the food item <NUM> in the image data captured by the imager <NUM>, the system <NUM> may update a usage or inventory of the food item <NUM> as being consumed or depleted. A food item property can be in the form of a fill level, an expiration date, a favorite, a recipe, a quantity, a brand, a condition, a placement, a name, a type, and the like, relating to the food item <NUM>. An object detection module may detect the location food item <NUM>.

<FIG> illustrates a portion of the interior <NUM> of the appliance <NUM>. The system <NUM> may be configured to determine a storage configuration <NUM> of the interior <NUM>. The storage configuration <NUM> can include the relative location of different food items <NUM> to one another, which can be tracked to create a full view of how food items <NUM> are stored in the appliance <NUM>, or refrigerator. Additionally, the system <NUM> can determine the location, identity, and size of the holders, such as the shelves <NUM>, compartments <NUM>, or drawers <NUM>, to further create the full view of the storage configuration <NUM>. In some examples, the storage configuration <NUM> of the appliance <NUM> can be cluttered with many food items <NUM>, such that a user viewing the interior <NUM> in real-time, or via a display (e.g. a mobile display) in communication with the vision system, may consider the interior <NUM> to be chaotic and messy.

In some examples, the system <NUM> can track and monitor the location, identity, and size of the food item <NUM> holders by way of a <NUM>-Dimensional coordinate system within the interior <NUM>. The system <NUM> may determine a unique set of coordinates for each shelf <NUM>, compartment <NUM>, drawer <NUM>, etc. located within the interior <NUM>. The set of coordinates may define the storage configuration <NUM>. Furthermore, the coordinates may be stored in a memory, or library, such that the system <NUM> can refer to the coordinates as needed.

The system <NUM> can be configured to provide an augmented reality view of the interior <NUM> in order to "clean-up", or organize, the view for the user on a display, such as a mobile device display or the appliance user interface, or display <NUM> (<FIG>), but is not limited to such. An augmented reality view can be in the form of an interactive experience of the real-word interior <NUM> of the appliance <NUM> where the storage configuration <NUM> and the food items <NUM> are "augmented" or "enhanced" by data provided by the system <NUM>. Turning to <FIG>, a mobile device <NUM> includes a display <NUM> on which a user can view an interior rendering <NUM> of the appliance <NUM> interior <NUM>. In this way, the interior rendering <NUM> is organized and more easily viewable than the interior <NUM> shown in <FIG>. Thus, a user may avoid "hunting" for food items <NUM> within the interior <NUM>.

The interior rendering <NUM> can be in the form of a digital rendering of the storage configuration <NUM> (<FIG>) which may include at least one digital food item representation <NUM>. The digital food item representation <NUM> may be in the form of a digital graphic, or graphical, image that corresponds to a food item <NUM>. As shown in <FIG>, the digital food item representation <NUM> corresponds to an apple stored on a digital representation of a shelf <NUM>. The interior rendering <NUM> may further include digital food item representations of multiple food items, such as a carton of a gallon of milk <NUM>, a lemon <NUM>, and a carton of eggs <NUM>, as well as a digital representation of the storage configuration <NUM> including digital representations of the shelves <NUM>, compartments <NUM>, and drawers <NUM>, including size and relative positions. Digital food item representations may include default images of an identified food item <NUM>, or images specific to a food item brand. Additionally, a digital food item representation <NUM> may include a single digital image for multiple identified food items <NUM>. For example, if the system <NUM> identifies a total of six yogurts, the interior rendering <NUM> may include a single digital image paired with a number to indicate the number of yogurts (<NUM>).

The system <NUM> may be configured to provide the interior rendering <NUM> in more than one layer such that the user may view the contents of the interior <NUM> over more than one view. Viewing the contents of the interior in layers may provide feedback that is easily viewable by the user. In some examples, the user may view a layer that corresponds with a selection of a food item property. The food item property may be determined by the object identification and detection modules based on one or more properties, characteristics, and location of a food item <NUM>.

Turning to <FIG>, in some examples, a user can select a food item property that corresponds to expiration dates of food items <NUM>. The interior rendering <NUM> can include at least one digital food item overlay <NUM> that corresponds to the selected food item property. Thus, in the event that a user selects expiration dates as a food item property for viewing, the digital food item overlay <NUM> may be in the form of a digital tag <NUM> including the expiration date of each food item <NUM>, which may also be represented by the digital food item representation <NUM>. The expiration date of a food item <NUM> may be determined by any suitable method, such as a user input, or identified by the vision system.

In another example, <FIG> illustrates the interior rendering <NUM> corresponding to a layer generated by a user selection of a fill level food item property. One or more digital fill level icons <NUM> can be displayed within the interior rendering <NUM> to indicate the fill level of a corresponding food item <NUM>. The fill level icon <NUM> can be in the form of any digital image, or overlay, suitable to indicate an approximate fill level. For example, the fill level icon <NUM> may be positioned adjacent the gallon of milk, represented by the digital food item representation <NUM>, and may include a level providing an approximate amount of milk remaining in the gallon container located within the interior <NUM> of the appliance <NUM>. The fill level, or usage, of a food item <NUM> may be determined by any suitable method, such as a user input, or identified by the vision system.

In yet another example, <FIG> illustrates the interior rendering <NUM> corresponding to a layer generated from a user selection of a user favorite's food item property. One or more digital favorite overlays <NUM> can be displayed within the interior rendering <NUM> to indicate that a specific food item <NUM> is a user favorite item. The digital favorite overlay <NUM> can be in the form of any digital image, overlay, or tag, suitable to indicate that a food item <NUM> is a favorite. For example, the digital favorite overlay <NUM> may be positioned adjacent a container of ice cream determined to be located within a freezer drawer indicated by the digital representation of the drawer <NUM>, and may include a tag displaying a heart icon for indicating that a user's favorite item, such as ice cream, is available within the interior <NUM> of the appliance <NUM>. The user favorite designation of a food item <NUM> may be determined by any suitable method, such as a user input, or identified by the vision system by user habits monitoring.

While food item properties selected by the user for augmented reality viewing generated by the system <NUM> have been described in relation to expiration dates, fill levels, and favorites corresponding to various food items <NUM>, the system <NUM> can generate an interior representation <NUM> corresponding to food items <NUM> according to a variety of determined characteristics of the food item <NUM>. In further examples, the system <NUM> can take into account the quality of the food item <NUM>, recipes, quantities, etc. It is also possible that the user selects more than one food item property such that multiple food item properties are represented in a single layer, or interior rendering <NUM>.

Referring now to <FIG> and <FIG>, block diagrams of exemplary embodiments of the storage system <NUM> are shown. The system <NUM> may comprise a controller <NUM> which may be integrated with or in communication with an appliance controller <NUM>. The controller <NUM> may comprise one or more processors and/or memory configured to complete the various processing steps and control procedures discussed herein. In an exemplary embodiment, the controller <NUM> may comprise an image processor <NUM> in communication with the at least one imager <NUM>. The image processor <NUM> may correspond to one or more microprocessors, discrete logic controllers or a combination of analog and/or digital circuitry. The image processor <NUM> may comprise a memory <NUM> and a communication circuit <NUM>.

The memory <NUM> may correspond to various forms of tangible or non-transitory machine-readable or computer-readable media. For example, the memory <NUM> may correspond to flash memory, random access memory (RAM), erasable programmable read only memory (EPROM), or other machine-readable media. The image processor <NUM> may store various image processing algorithms, control instructions, and additional information in the memory <NUM> to complete one or more of the processing tasks described herein.

The communication circuit <NUM> may correspond to a wired or wireless communication device through which the controller <NUM> may communicate with and/or access various forms of information. For example, the controller <NUM> may access or update an inventory or grocery database <NUM> via the communication circuit <NUM>. The inventory or grocery database <NUM> may correspond to a remote or cloud-based storage that may be accessible by a variety of devices to access the inventory and/or grocery database information as described herein. Additionally, the communication circuit <NUM> may further be in communication with an additional remote server or database <NUM> from which controller software updates, object recognition information, algorithms, object libraries, recipe libraries, and various additional information for use with the system <NUM> may be accessed.

Though discussed in reference to remote or cloud-based databases, the databases <NUM> and/or <NUM> may also correspond to local storage provided in a memory of the controller <NUM>. Additionally, in some embodiments, a portion of each of the databases <NUM> and <NUM> may be stored in the local memory while additional portions may be stored in the remote databases <NUM> and <NUM> in communication with the controller <NUM> via the communication circuit <NUM>. The communication circuit <NUM> may utilize various forms of communication interfaces including, but not limited to, the following wireless communication protocols: <NUM>, <NUM>, Wi-Fi ®, Wi-Max®, CDMA, GSM, and/or any suitable wireless communication protocol.

As discussed herein, the system <NUM> may be operable to complete various image processing and identification steps corresponding to the food item stored within the interior <NUM>. Such detection, recognition, and/or identification may be completed by the image processor <NUM> by processing the image data received from the at least one imager <NUM>. In order to process the image data, the image processor <NUM> may comprise a plurality of processing modules configured to improve or simplify the image data, and/or determine an identity of the food item <NUM>. In this way, the image processor <NUM> may provide for the determination of the identity, quantity, and/or proportion of the food item <NUM>.

In an exemplary embodiment, image processor <NUM> may comprise an image sampler <NUM>. The image sampler <NUM> may gather a sequence of image frames from the image data received from the at least one imager <NUM>. The sequence of image frames may be temporarily stored in a frame buffer <NUM>. From the frame buffer <NUM>, the image data from the image frames may be retrieved by a pre-processor <NUM>. The pre-processor <NUM> may process the sequence of image frames from the image data to enhance the contrast, adjust a rotation, correct a boundary, and/or scale or crop the image data. The pre-processor <NUM> may accomplish such steps by processing the image data with a variety of enhancement techniques, including, but not limited to, histogram equalization, image segmentation, filtering, etc..

Enhanced image data may be communicated from the pre-processor <NUM> to the object detection module <NUM>. The object detection module <NUM> may process the enhanced image data corresponding to the location of one or more food items <NUM> as described herein. The image data may be communicated to an object identification module <NUM>. The object identification module <NUM> may process the image data to determine an identity, type, category, proportion, and/or quantity of the food item <NUM>.

The object identification module <NUM> may utilize a variety of scanning methods and/or algorithms to determine the identity of the food item <NUM>. Such algorithms may include, but are not limited to, character recognition, feature extraction, clustering, pattern matching, neural network processing, optical character recognition (OCR), or various other recognition techniques. In some embodiments, the object identification module <NUM> may further be in communication with an object database <NUM>. The object database <NUM> may comprise a library of features and/or characteristics for comparison to the enhanced image data by the object identification module <NUM> in order to determine the identity of the food item <NUM>. In this way, the system <NUM> may be operable to determine the identity of the food item <NUM> as discussed herein.

Although the image processor <NUM> is demonstrated as being an integral component of the controller <NUM>, the image processor <NUM> may correspond to a remote image processing system that may be in communication with the controller <NUM> via the communication circuit <NUM>. In this way, the storage system <NUM> may communicate the image data to a remote image processor via the communication circuit <NUM> in order to determine the identity of the food item <NUM> with the object identification module <NUM>. Upon determination of the identity of the food item <NUM> from the image data, the remote image processor may provide an indication of the identity and any other relevant information to the controller <NUM> via the communication circuit <NUM>. Accordingly, though the storage system <NUM> is discussed in reference to specific examples, the functions described in reference to the system <NUM> may be completed in a variety of ways without departing from the spirit of the disclosure.

In some embodiments, the at least one imager <NUM> may correspond to a plurality of imaging devices or a stereoscopic imaging device. In this way, the image processor <NUM> may also be operable to determine a position and proximity of the food item <NUM> based on depth data gathered by the stereoscopic imaging devices. Such positional information of the food item <NUM> may be beneficial in identifying or distinguishing a location of the food item <NUM>. Accordingly, the storage system <NUM> may correspond to a scalable system operable to suit a variety of applications. A stereoscopic system may also be used to estimate volume of a food item <NUM>. In some embodiments, the at least one imager <NUM> may correspond to an infrared imaging device, thermal imaging device, or a combination of thermal and conventional imaging devices. The thermal imaging device may correspond to a focal plane array (FPA) utilizing microbolometers as FPA sensors. Accordingly, the image data captured by the system <NUM> may comprise thermal image data and/or conventional image data in the visible light spectrum. In some embodiments, the system <NUM> may be operable to distinguish between a plurality of objects having a like visual appearance in the image, but differing temperatures. For example, in some embodiments, the system <NUM> may be operable to track an inventory of an object that is removed from the interior <NUM> at a cold temperature and replaced in the operating volume at a warm temperature. Accordingly, by utilizing the thermal imaging data, the system <NUM> may be operable to distinguish additional status information for the food item <NUM>.

Referring to <FIG>, <FIG>, and <FIG>, the controller <NUM> of the system <NUM> is configured to determine a query from the user regarding the food item <NUM>. A query from the user may be in the form of a gesture, a vocal communication, or any suitable haptic command. The controller <NUM> is configured to determine a response to the query, which is based on an identity or location of the food item <NUM>. A response to the query may include a voice message, selective lighting, etc. Examples of selective lighting can include, but are not limited to, a pattern of light <NUM> (<FIG>) including a light spot on the food item <NUM>, a light color on the food item <NUM>, or words or graphics generated by an arrangement of pixels, a pulsing light, dimming light, or lighting a specific zone of the interior <NUM>.

In some examples, a user may query the system <NUM> with a query that includes a voice message inquiring about an optimal storage location for the food item <NUM>, which may include, "Where do I store the butter"? The controller <NUM> may analyze the query and determine the response for guiding the user for storage of the food item <NUM>. For example, the system <NUM> may then provide a pattern of light <NUM> that can include selective lighting of at least a portion of the appliance <NUM>, such as a compartment <NUM>, to provide an indication of the optimal storage location for the butter, or food item <NUM>.

In another example, a user may query the system <NUM> with a query that includes a voice message inquiring about the location of a food item <NUM>, which may include, "Where are my strawberries"? The controller <NUM> may analyze the query and determine the response for guiding the user to the location of the food item <NUM>. For example, the system <NUM> may then provide a pattern of light <NUM> that can include a beam of light directed to the strawberries, or food item <NUM>, to provide an indication of the location. The beam of light may include colors that correlate to a determined food item property, such as a quality of the food. In some examples, a red beam of light may indicate that the food item <NUM> has expired, or contains mold, while a green beam of light may indicate that the food item <NUM> is unexpired, or fresh.

In yet another example, a user may query the system <NUM> with a query that includes a voice message inquiring about available ingredients for a recipe, which may include, "What do I have for making a sandwich"? The controller may analyze the query and determine the response for providing feedback to the user regarding multiple food items <NUM> available for a recipe. For example, the system <NUM> may then provide a pattern of light <NUM> that can include a beam of light directed to each of the food items <NUM>, which may include bread, meat, vegetables, and condiments stored within the interior <NUM> to provide an indication of available food items <NUM> for making a sandwich.

While responses by the system <NUM> have been described in relation to storage or recipes for the food item <NUM>, the system <NUM> can determine responses to a user query regarding a food item <NUM> to provide assistance to the user using a variety of determined characteristics of the food item <NUM>. In further examples, the system <NUM> can take into account the status of the food item <NUM>, which can include a depletion/usage for the food item <NUM> and determine a fill level.

A response includes a heat map of the interior <NUM> generated by the lighting system. The heat map includes lighting up the interior <NUM> according to variations in temperature that correspond to variations in colored light. Furthermore, the variations in colored light indicate poor placement of food items <NUM>. For example, placing yogurt towards the rear, or back, of the interior, can cause the yogurt to freeze. Poor placement may be indicated by red colored light, while optimal placement may be indicated by green colored light. Thus, the heat map may include multiple light projections, or beams throughout the interior <NUM>. Additionally, variations in colored light indicate issues within the interior <NUM>. Messes may result from placing a large frozen mass, such as a frozen turkey, within the interior <NUM> where humidity may increase and lead to dripping. The heat map indicates if dripping has occurred.

<FIG> illustrates a flow chart demonstrating a process <NUM> wherein a user queries the system <NUM> in regards to a food item <NUM>. The process <NUM> may begin at step <NUM> when the system <NUM> detects a user query regarding a food item. The user query can pertain to any food item property that can be detected or determined by the system <NUM> for the food item <NUM>. At step <NUM>, the controller <NUM> of the system <NUM> may analyze, or process, the query and then determine the appropriate response at step <NUM>. The appropriate response at step <NUM> can include feedback to the user regarding the food item <NUM>, which may include any suitable audio or visual techniques. At step <NUM>, the controller <NUM> of the system <NUM> can initiate the response. Initiating the response can include activating the light source <NUM> (<FIG>). Following the initiating of the response at step <NUM>, the system <NUM> can provide the response and end the process <NUM> at step <NUM>.

Providing the response can include projecting the response onto the food item <NUM>. Projecting the response onto the food item <NUM> can include any suitable lighting technique to provide an indication, including, but not limited to: words, icons, graphics, animations, light beams, diffused light, and coloration of light, and any combination thereof. For example, the projection <NUM> (<FIG>) may include an expiration date. The expiration date can be in the form of colored pixels where the coloring may provide an indication if the expiration date has passed or not, such as green for not expired and red for expired.

While the method <NUM> is described as including steps <NUM>-<NUM>, it is possible for the method <NUM> to include additional or fewer steps. Additional steps can include any suitable step or process as described herein. In some examples, the method <NUM> may include analyzing more than one query from the user and determining more than one response. Furthermore, initiating the response at step <NUM> may occur simultaneously with providing the response at step <NUM>.

According to one aspect of the present disclosure, a food storage system includes an appliance having a cabinet defining an interior. The interior includes at least one of a shelf, compartment, or drawer configured to store at least one food item. A vision system includes at least one imager configured to record image data related to the at least one food item stored within the interior. A controller is in communication with the vision system. The controller is operable to determine an identity of the least one food item, determine a storage configuration of the interior, generate an augmented reality representation of the interior based on the determined identity of the at least one food item and the storage configuration of the interior and display the augmented reality representation. The augmented reality representation includes a digital rendering of the storage configuration and a digital food item representation corresponding to the at least one food item is arranged within the digital rendering.

According to another aspect of the present disclosure, determining the storage configuration of the interior includes determining the position and size of the at least one of a shelf, compartment, and drawer within the interior.

According to another aspect of the present disclosure, the controller is in communication with a display for at least one of a mobile device and the appliance and the augmented reality representation is displayed on the display.

According to another aspect of the present disclosure, the augmented reality representation includes at least one digital food item overlay corresponding to a food item property of the at least one food item.

According to another aspect of the present disclosure, the food item property is a fill level and the digital food item overlay is a graphical image configured to indicate the fill level of the at least one food item.

According to another aspect of the present disclosure, the food item property is an expiration date and the digital food item overlay is a graphical image configured to indicate the expiration date of the at least one food item.

According to another aspect of the present disclosure, wherein the food item property is a favorite and the digital food item overlay is a graphical image configured to indicate that the at least one food item is a favorite.

According to one aspect of the present disclosure, a food storage system includes an appliance having a cabinet defining an interior. The interior includes at least one of a shelf, compartment, or drawer configured to store at least one food item. A vision system includes at least one imager configured to record image data related to the at least one food item stored within the interior. A lighting system includes at least one light source configured to project a pattern of light. A controller is in communication with the vision system. The controller is operable to determine an identity and a location of the at least one food item, analyze a query from a user regarding the at least one food item and control the lighting system to project a pattern of light onto the at least one food item in response to the query. The pattern of light provides feedback to the user indicating at least one of a location or a food item property of the at least one food item.

According to another aspect of the present disclosure, the pattern of light includes words.

According to another aspect of the present disclosure, the pattern of light includes a color corresponding to a temperature of the at least one food item.

According to another aspect of the present disclosure, the pattern of light includes a color corresponding to a condition of the at least one food item.

According to one aspect of the present disclosure, a food storage system includes an appliance having a cabinet defining an interior. The interior includes at least one of a shelf, compartment, or drawer configured to store at least one food item. A vision system includes at least one imager configured to record image data related to the at least one food item stored within the interior. A controller is in communication with the vision system. The controller is operable to detect at least one food item property for the at least one food item, generate an augmented reality representation of the interior based on a user selection of a food item property and display the augmented reality representation. The augmented reality representation includes at least one digital food item overlay corresponding to the food item property.

According to another aspect of the present disclosure, the food item property is a fill level and the digital food item overlay is a graphical image configured to indicate the fill level of a corresponding food item.

According to another aspect of the present disclosure, the food item property is an expiration date and the digital food item overlay is a graphical image configured to indicate the expiration date of a corresponding food item.

According to another aspect of the present disclosure, the food item property is a favorite and the digital food item overlay is a graphical image configured to indicate at least one favorite food item of a user.

According to another aspect of the present disclosure, displaying the augmented reality representation includes displaying multiple digital food item overlays corresponding to multiple food items.

According to another aspect of the present disclosure, the food item property is a recipe and the digital food item overlay is multiple graphical images configured to indicate all food items within the interior necessary for the recipe.

According to another aspect of the present disclosure, the food item property is a food item condition and the digital food item overlay is a graphical image configured to indicate a food item condition.

According to another aspect of the present disclosure, the augmented reality representation further comprises at least one digital food item representation of the at least one food item.

According to another aspect of the present disclosure, the controller is further operable to determine a storage configuration of the interior, generate an augmented reality storage representation of the interior based on the storage configuration of the interior and display the augmented reality storage representation.

According to another aspect of the present disclosure, the augmented reality storage representation includes a digital rendering of the storage configuration.

According to another aspect of the present disclosure, determining the storage configuration of the interior includes determining the position and size of the at least one of a shelf, compartment, or drawer within the interior.

According to another aspect of the present disclosure, the controller is further operable to determine an identity of the least one food item, generate a digital food item representation corresponding to the at least one food item based on the determined identity of the at least one food item and display the digital food item representation.

According to another aspect of the present disclosure, the controller is further operable to analyze a query from a user regarding the at least one food item control a lighting system to project a pattern of light onto the at least one food item in response to the query and the pattern of light provides feedback to the user indicating at least one of a location or a food item property of the at least one food item.

Claim 1:
A food storage system (<NUM>) comprising:
an appliance (<NUM>) having a cabinet (<NUM>) defining an interior (<NUM>), the interior (<NUM>) comprising at least one of a shelf (<NUM>), compartment (<NUM>), or drawer (<NUM>) configured to store at least one food item (<NUM>);
a vision system comprising at least one imager (<NUM>) configured to record image data related to the at least one food item (<NUM>) stored within the interior (<NUM>); and
a controller (<NUM>) in communication with the vision system, wherein the controller (<NUM>) is operable to:
detect at least one food item property for the at least one food item (<NUM>);
generate an augmented reality representation of the interior (<NUM>) based on a user selection of a food item property;
display the augmented reality representation;
wherein the augmented reality representation comprises at least one digital food item overlay (<NUM>, <NUM>, <NUM>) corresponding to the food item property;
wherein the controller (<NUM>) is configured to determine:
a query from the user regarding the food item (<NUM>) and
a response to the query based on an identity or location of the food item (<NUM>);
characterised in that the interior (<NUM>) includes a lighting system including at least one light source (<NUM>), the at least one light source (<NUM>) being configured to provide selective lighting;
wherein the response includes a heat map of the interior (<NUM>) generated by the lighting system;
wherein the heat map includes lighting up the interior (<NUM>) according to variations in temperature that correspond to variations in colored light;
wherein variations in colored light further indicates poor placement of food items (<NUM>); and
wherein, additionally, variations in colored light indicate issues within the interior (<NUM>), the heat map indicating if dripping has occurred.